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Executive summary 


The project 


Act, Sing, Play (ASP) offered music and drama tuition to Year 2 pupils. The aim of the programme was to 
evaluate whether music workshops had a bigger impact than drama workshops in terms of pupils’ maths 
and literacy attainment. The evaluation was based on the hypothesis that participation in high-quality 
music instruction promotes educational attainment over and above instruction in other artistic pursuits 
(see Schellenberg, 2004). 


The ASP programme was developed specifically for this trial and ran from September 2013 to June 2014: 
909 pupils participated in 19 schools across London, Essex, Sussex and Coventry. In each participating 
Year 2 class, pupils were randomly allocated to one of three groups: violin or cello workshops (ASP- 
strings), singing lessons (ASP-singing), or drama workshops (ASP-drama). The two music groups (strings 
and singing) represented the treatment arms of the trial. Students in the ASP-drama workshops 
represented the control. Each workshop had around 10 students. Workshops were held once a week over 
32 weeks. The programme was delivered by Creative Futures, funded by the Education Endowment 
Foundation, and independently evaluated by NatCen. 


Key Conclusions 


1. This evaluation provides no evidence that ASP-music workshops had a greater impact on 
maths or literacy attainment than ASP-drama workshops. 


. Analysis of students receiving free school meals similarly found no evidence that ASP-music 
workshops had a greater impact on maths or literacy than ASP-drama workshops. 


3. The process evaluation suggested that some tutors—particularly those with less experience of 
teaching groups of primary school children—needed more guidance on how to run their 
sessions. 

4. Although not necessarily typical, there were related concerns that some strings workshops 
struggled to keep students focussed on learning music. 


5. Class teachers reported that confidence and social skills had improved for some pupils. 
Teachers also felt it was important that children from disadvantaged backgrounds had the 
opportunity to learn a new skill that they might otherwise not be able to access. 


: : Security rating awarded as part 
Security rating of the EEF peer review process 


Findings from this trial have a high degree of security. The trial was set up as a randomised control trial, 
with pupil-level randomisation. The trial was classified as an effectiveness trial, meaning that it sought to 
test whether the intervention can work at scale, and in situations where the developers are no longer the 
only deliverers. Before the trial started, there was a good balance of pupil characteristics between the 
group receiving music workshops and those receiving drama workshops (the control group). The attrition 
was low: in total, results from 814 pupils were assessed compared to 909 pupils who began the project 
(attrition of 10%). There was no evidence that systematic attrition had biased the results. There were 
some minor threats reported to the validity of intervention protocol. The security rating of 4 padlocks fits 
with the overall output of this report. 
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Results 


e This evaluation found no evidence that the ASP-music workshops had a greater impact than the 
ASP-drama workshops in terms of maths or literacy attainment. Although small effects (positive 
and negative) were identified during the analysis, it could not be concluded that these effects 
were anything other than random chance. Each of the two different types of music workshop 
(strings and singing) was also analysed separately. As was the case with the main analysis, 
neither strings nor singing workshops showed evidence of out-performing drama workshops in 
advancing maths or literacy attainment. 


e Similarly, there was no evidence that, among pupils eligible for free school meals (FSM), music 
workshops had an impact on attainment when compared to the control group (ASP-drama). 
Again, effect sizes were close to zero, indicating that disadvantaged children did not particularly 
benefit from music instruction relative to drama workshops. 


e Interviews with school staff and workshop tutors suggested that there were differences in the way 
the programme was implemented by different tutors, reflecting the flexible nature of the 
programme. Tutors valued the ability to choose flexible approaches to music tuition. However, the 
process evaluation suggested that some tutors—particularly those with less experience of 
teaching groups of primary school children—needed more guidance on how to run their sessions. 
This led to related concerns that in some of the strings workshops, it was a struggle to keep 
students on task. 


e Ingeneral, the process evaluation suggested that pupils enjoyed participating in the program, and 
were engaged. Teachers also reported that some pupils’ confidence and social-skills improved 
during the program. 


e Last, it is important to reiterate that the design of this trial only allows for comparisons between 
music and drama workshops. This design was chosen for several reasons, including the fact that 
a previous study (Schellenberg, 2004) had found that music tuition increased attainment over and 
above other creative pursuits. This present study, which involved a much larger sample than 
Schellenberg, was unable to replicate this result. 


Cost 


The Act, Sing, Play programme had financial costs relating to tutor training, travel, purchase of musical 
instruments, and other resources associated with delivering the programme. Overall, the total financial 
cost to schools, for the programme to be delivered for one academic year, was £5,980 per class. This 
equates to around £219 per pupil per year. Costs in subsequent years will be less as the cost of musical 
instruments would not be repeated every year. The program imposed a low cost in terms of teacher time. 
In many instances, when children were at the workshops, teachers gained a period to focus on other 
responsibilities. 


Effect size Progress Security Cost 
(95% Cl) compared Rating of music 
to drama Ne); <=1 ale) @) 


iS 
MUSIC workshops 0.003 0 months 
compared to (-0.01 to 0.10) 
drama workshops 
Literacy* MUSIC workshops 0.03 0 months 
compared to (-0.07 to 0.13) 
drama workshops 


*Standardised PIPs score. 
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Introduction 


The Act, Sing, Play programme 


Act, Sing, Play was a program that offered music and drama tuition to Year 2 pupils. It was modelled ona 
similar, although smaller, study by Schellenberg in Toronto in 2004. The programme was delivered by 
Creative Futures (CF)* a multi-arts organisation which uses active participation in creativity and the arts to 
enhance aspiration, achievement and quality of life for children. It delivers programmes in primary and 
secondary schools as well as in local communities, early years’ settings, and with looked-after children. 


The Act, Sing, Play programme was developed specifically for this trial by Creative Futures and ran from 
September 2013 to June 2014 in primary schools from four areas of England: the London Borough of 
Hounslow, East Sussex, Coventry, and Essex. In most cases schools were recruited in collaboration with 
local music education hubs—a nationwide network of 122 schools, music organisations, and arts 
organisations overseen by Arts Council England on behalf of the Department for Education. Class groups 
of around 30 pupils were randomly split into three arts workshop groups of approximately 10 pupils each. 
One group received stringed instrument lessons (violin or cello), the second group received singing 
lessons based loosely on the ‘Kodaly’ method (more detail about this method can be found within the 
intervention section below), and the third was an ‘active’ control group participating in drama lessons. 


All three sessions ran concurrently, within school hours, and lasted for 45 minutes. All sessions were led 
by freelance tutors provided by Creative Futures or by local music education hubs. Although tutors in 
each arts group were given a broad curriculum to follow, the specific content and format of each session 
was left to individual tutors. More description of the curriculum can be found within the ‘Curriculum’ 
section below and in the Appendices. 


For the strings sessions, violins and cellos were loaned to schools by the local music services for the 
duration of the programme. The delivery team encouraged schools to allow these instruments to be taken 
home for practice, although in most cases this did not happen. This is discussed further in the 
‘Instruments and Practicing’ section below. 


Background evidence 


There is consistent evidence that children from economically disadvantaged backgrounds have less 
access to music, art, and out of school activities than their more advantaged peers (Etsyn, 2012; Gorard, 
Huat See and Davies, 2012). Analysis of the Millennium Cohort Study showed that 26% of 11-year-olds 
from families above the poverty line had private music lessons compared to 4% of children in poverty 
(Chanfreau and Tanner, 2014). 


Related to the issue of access and fairness in receiving music tuition is the question of whether listening 
to music or playing an instrument can have a positive effect on cognitive ability. A number of 
observational studies have reported correlations between music and attainment. In particular, a meta- 
evaluation undertaken by Winner and Cooper (2000) to quantify the results of existing research between 
1950 and 1998 found a significant positive relationship between arts education and academic 
achievement. However, this research examined the effects of arts in general, rather than music 
specifically. Moreover, it could not conclude that the relationship was necessarily causal since there were 
no controls for selection bias. It could simply be the case that high achievers were more likely to study 
arts subjects. The two studies examined by Winner and Cooper that had experimental designs with 
randomised assignment showed negligible effect sizes. 


‘Creative Futures have also engaged the Institute of Education (loE) to conduct a parallel evaluation of 
Act Sing Play's impacts on musical progression and participating pupils’ self-efficacy and learner identity. 
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Evaluation research has identified that music interventions have positive impacts on academic attainment, 
particularly for younger children of primary school age (Sutton Trust-EEF Teaching and Learning Toolkit, 
2014). A meta-evaluation by Standley (2008) found that when music and reading activities were 
combined, the benefits for children in terms of recognising the alphabet and phonetic patterns were 
substantial. Lonie’s study (2010) also found links between ‘music making’ in the early years and 
improvements in reading and language skills. Research in the Sutton Trust toolkit also highlights other 
wide-ranging benefits from participation in music activities including positive attitudes and wellbeing, and, 
for young children, improved communication and relationships with parents. 


One of the most relevant studies to this evaluation was undertaken by Schellenberg (2004) who set out to 
test the hypothesis that ‘music makes you smarter’. The study looked at the effect of 36 weeks of music 
tuition on the IQ of six-year-olds: 144 children were randomly assigned to receive keyboard, singing 
(using Kodaly vocal instruction), drama, or no arts lessons. Schellenberg concluded that results indicated 
there was evidence that music lessons caused increases in |Q. However, Steele (2005) questioned the 
theoretical basis of the grouping, particularly because the Kodaly method was presented as being very 
different from standard musical instruction. Steele’s reanalysis of Schellenberg’s data, in which singing 
and music were examined separately, found that neither type of musical intervention was significantly 
different from the control group’s outcomes. Other limitations of Schellenberg’s (2004) research include 
the self-selection of children into the study by parents via an opt-in process following an advert in a local 
newspaper, and the small size for the trial (n = 132 for analysis). 


The Act, Sing, Play programme builds on research already undertaken by using a larger sample size and 
focusing on disadvantaged pupils to assess whether music tuition affects maths and literacy attainment in 
the short-term for similarly-aged children. 


Evaluation objectives 


The aim of the impact evaluation was to measure the impact of the music components of the Act, Sing, 
Play programme, as delivered by Creative Futures, on the maths and literacy attainment of participating 
pupils, a subgroup of whom were eligible to receive Free School Meals. 


The aims of the process evaluation were to provide a detailed understanding of how the intervention 
programme was delivered in practice, and to identify which elements were perceived as important to its 
success as well as any barriers schools faced in implementing the programme. This evidence was 
collected for two main reasons: 


e to help understand and explain the impact results; and 


e to inform decisions about whether and how to scale-up or roll-out the Creative Futures 
programme more widely. 


Project team 


The Act, Sing, Play programme was overseen by Julian Knight and Vanessa Stansall from Creative 
Futures alongside Jo Saunders from the Institute of Education (loE). 


The programme was delivered in schools by freelance tutors. There were 32 tutors in total who were 
employed either by Creative Futures directly or by their local music services with whom Creative Futures 
had a contract. At the start of the programme, 13 strings tutors, 9 vocal tutors and 10 drama tutors were 
employed. Throughout the programme some tutors taught at more than one school and two of the tutors 
taught both singing and strings sessions. 


There were three additional tutors brought in after the first term to replace two who had gone on maternity 
leave and one who had left. Although all music tutors had previous instrument teaching experience, their 
level of classroom teaching experience varied. 
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The evaluation team at NatCen Social Research was led by Cheryl Lloyd and Amy Skipp. Gemma Lewis 
and Natasha Reilly were responsible for co-ordinating pre- and post-intervention testing which was carried 
out by NatCen interviewers. The impact evaluation was designed by Stephen Morris and the analysis was 
carried out by Zsolt Kiss and Julia Griggs. The process evaluation work was led by Sarah Haywood, 
assisted by Natasha Reilly. 


Ethical review 


NatCen Social Research obtained ethical approval for all elements of the evaluation from its internal 
Research Ethics Committee on 8 April 2013. This approval covered the processes for testing pupils and 
analysing the results, NPD linkage, communicating with and carrying out interviews with school staff and 
tutors, as well as carrying out observations of music lessons within schools. 


Schools each signed a copy of aMemorandum of Understanding. This outlined what their participation in 
the study entailed, including information about their involvement in the testing of pupils and interviews and 
observations. A copy of this can be found in Appendix G and copies of the consent form for parents can 
be found in Appendix E. 
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Methodology 


Trial design 


The Act Sing Play (ASP) evaluation was designed as a three-arm pupil-level randomised controlled trial of 
the impact of music workshops on the attainment of Year 2 students. Pupil-level randomisation was 
chosen to increase power and reduce costs. The trial design required students to be randomly allocated 
to one of three groups within classes: one group were allocated to receive lessons on violin or cello (ASP- 
strings) while a second were allocated to receive singing lessons (ASP-singing). These two groups 
represented the treatment arms. A difficult decision then had to be made as to what the pupils in the 
control arm would do while their classmates had music workshops. Drama workshops (ASP-drama) were 
ultimately chosen so that all pupils in the control arm were occupied with a known activity that did not 
have a clear impact on attainment (which would have been the case if the control group had stayed back 
in their classes and received small-group tuition). 


Pupils were assigned to groups on a 1:1:1 basis. Pupil-level randomisation was stratified within class by 

eligibility for Free School Meals (FSM), sex and date of birth (See randomisation, below). This stratified 
randomisation took place within classes in participating primary schools. It should be noted that the within 
class, pupil-level randomisation meant there was potential for ‘contamination’ to occur between pupils 
belonging to the same class, but assigned to different study arms (meaning that children assigned to 
different study arms may share the content of their lessons/workshops with classmates and thereby 
reduce the impact of the intervention). 


It was anticipated that this design would help to reduce some of the potential consequences of ‘resentful 
demoralisation’ and ‘compensatory rivalry’ when knowledge of allocation to a ‘non-active’ control group 
became known, as well as alleviating problems associated with ‘novelty bias’ (Hawthorne Effects). 


An active control also makes treatment contrasts explicit through providing the controls with a specified 
‘activity’. However, it is important to acknowledge that parents may compensate for allocation to (what is 
perceived to be) a less-preferred treatment by acquiring preferred treatments independently as a 
consequence of the study. 


Due to the absence of an ‘inactive’ control group, comparisons have been made between two forms of 
musical tuition relative to drama. The effects of the treatments have been estimated by comparing 
average outcomes between the three intervention arms at post-test. 


School eligibility 


Primary schools were recruited for the intervention by the Creative Futures delivery team. Recruitment 
took place in four areas of the UK: Coventry, East Sussex, Hounslow and Essex. In most cases schools 
were recruited in collaboration with local music education hubs,’ with the aim of including schools with a 
diverse range of characteristics based on size, urban-rural location, and levels of deprivation. The 
delivery team also took into account the level of prior involvement in music provision to help with the aim 
of using the programme to build stronger links between the school and the local Music Education Hub. 
Only schools with no current music instrument provision for Year 2 pupils, and no musical instrument 
teaching in Year 1, were eligible to participate. 


? Music Education Hubs are funded and overseen by Arts Council England on behalf of the Department 
for Education, and represent a new way of organising and delivering music education to children and 
young people. A nationwide network of 122 hubs has been set up and includes schools, professional 
music organisations and arts organisations. Hubs work in local areas to bring stakeholders together to 
facilitate co-ordinated music education provision. http://www.artscouncil.org.uk/news/arts-council- 
news/we-announce-successful-music-education-hub-applica/ 
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Five local Music Education Hubs were approached about the study. Nineteen schools originally agreed to 
participate in the Act, Sing, Play programme, however following the initial recruitment process two schools 
decided that they no longer wished to participate in the study. The delivery team recruited a further two 
schools as replacements. This change in participating schools took place before pupil pre-intervention 
testing and randomisation were carried out, and before any Memoranda of Understanding (MOUs) were 
signed. Therefore, pre-test data was collected from the replacement schools at the same time as all other 
schools meaning that the change should not affect results. 


Pupil eligibility 


All Year 2 pupils in participating schools were eligible to take part in the programme. Whilst parental 
consent was sought prior to the pre-test and randomisation, no opt-outs were received (see Pupil 
Recruitment, below). At the pre-test, outcome measures were collected from each child and demographic 
information was provided by the child’s school. After completion of the pupil Performance Indicators in 
Primary Schools (PIPS) assessments, pupil test scores were made available to the evaluation team on 
the Centre for Evaluation and Monitoring’s (CEM) PIPS+ secure website (See Outcomes, below). 


Pupils with Special Educational Needs 


The delivery team worked with schools to identify pupils for whom certain sessions would not be 
appropriate. These pupils were excluded from the randomisation but took part in the arts workshop 
session considered to be most appropriate by their teacher, in consultation with the delivery team. 
Schools were asked that if pupils had Special Needs Assistants (SNAs) assigned to them that they should 
remain with pupils for the workshops. 


Intervention 


The Act, Sing, Play Programme took place in 19 primary schools in England between mid-September 
2013 and July 2014. Thirty-two workshops took place throughout the school year with Year 2 pupils. The 
schedule for sessions was developed through a combination of tutor availability and school preference, 
including the availability of the necessary rooms. All workshops took place during school hours, and most 
schools opted for the workshops to take place in the afternoons. 


The project used a three arm (two treatment groups and an active control group) research design 
meaning that all Year 2 pupils taking part within each school were assigned to one of three groups: 


e Group 1 were taught singing using a curriculum loosely based on the Kodaly Method. This 
curriculum aimed to give pupils a thorough musical grounding through rhythm and singing. 


e Group 2 were taught to play a string instrument (violin or cello). The curriculum assumed that all 
pupils were beginners and took them through the first stages of learning the violin or cello. 


e Group 3 were an ‘active’ control group participating in drama sessions. This curriculum aimed 
to develop pupils’ imaginations through story-telling. 


More information on each of the curriculums can be found within the Curriculum section below. 


Each arts group was intended to have a maximum of ten pupils, although some were larger (up to 15) due 
to the large class sizes in some schools. All three sessions ran at the same time and lasted for 45 
minutes. For timetabling reasons, there was variation between schools as to what time of day sessions 
were held. 


° More information about the Kodaly Method can be found at 
www.britishkodalyacademy.org/kodaly_approach.htm 


Education Endowment Foundation 9 


Creative Futures: Act, Sing, Play 


Class teacher involvement 


Class teachers were not required to attend the workshops, however, they were asked to attend the first 
string instrument workshop to help tutors with the allocation and labelling of instruments. Teachers were 
also encouraged to participate in or observe any of the workshops if they chose to. 


In the event of a tutor absence, in the first instance the programme team tried to find a replacement tutor; 
if this was not possible, the class teacher was required to take the group whose tutor was absent and the 
other two groups continued as usual. 


Instruments 


Violins and cellos were loaned to schools by the local music services for the duration of the Act, Sing, 
Play programme. 


The delivery team encouraged schools to allow the instruments to be taken home for practice, however 
instruments which were taken home by pupils needed to be underwritten by parents, or the school, in 
case of loss or damage. 


Schools were asked to provide six music stands for the strings group. Some strings and singing tutors 
also used pianos belonging to the school during workshop sessions. 


Training 


Prior to the start of the sessions, tutors from all three arts groups (strings, singing and drama) attended a 
three-hour training session together. The training session included a project overview; a presentation 
around the key elements of a high quality session and how to set and measure learning objectives; and 
break-out sessions for the 3 different activities to go through the curriculum objectives for each subject led 
by the consultant who had devised each curriculum. In addition, vocal tutors attended a further session in 
December 2014. This was described as more of a ‘workshop’ than formal training and focused on the new 
curriculum elements for the forthcoming term, and sharing best practice from the previous term. One 
drama and one strings tutor who could not attend any sessions were sent resources and training videos 
by email. The three tutors who started on the programme late did not receive formal training; they had an 
individual introduction to the project, meeting with the project team, and were offered support throughout 
the programme from the curriculum lead for their subject area. 


The curriculum 


Tutors in each arts group were given guidance on the curriculum content and the format of workshop 
delivery: this guidance was relatively ‘loose’ and far from prescriptive about either the approach to take in 
each of the sessions, or how to achieve specified outcomes. The content and format of sessions was very 
much down to the individual tutors. The following sections outline the curricula for each arts group. 


Strings 


Given the range of string teaching methods, and taking into account the expertise of individual tutors, it 
was not considered appropriate to impose a string teaching ‘method’. Instead, strings tutors were givena 
planning document which stated that, although they were free to use their own teaching methods 
throughout the trial, some elements needed to be consistent. It contained: 


e reference to ‘Fiddle Time Starters’, a beginner’s violin book, a copy of which was also 
provided for every student; 


e aset of termly milestones which tutors were expected to work towards—these included 
specific milestones such as ‘by the end of term one, students should be able to use different 
bow lengths and bow speeds’; and 
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e reference to some of the common exercise/activity types: for some weeks these activities 
were specified within the lesson plan, but not all. 
The delivery team also provided tutors with a pack of activities as a starting point which tutors were told 
they may wish to supplement with other activities from the ‘Fiddle Time Starters’ book or their own 
materials. 


The strings curriculum included a number of milestones that pupils were required to reach each term: 


Term 1: 


e be able to demonstrate how to take the instrument out of the case and put it back; 
e be able to demonstrate good playing posture, and how to hold the instrument; 

e be able to demonstrate how to hold the bow; 

e be able to recognise the symbols for, and be able to play, up bows and down bows; 
e be able to use different bow lengths and bow speeds; 

e be able to pluck strings including string crossing; 

e be able to recognise and play— 

e —crotchets 

e quavers 

e = minims 

e crotchet rests; 

e understand simple double, triple and quadruple time signatures; 

e know the names of the open strings and be able to read them on a stave; 

e understand the terms ‘pizzicato’, ‘arco’, ‘col legno and ‘tremolo’; and 

e be able to play with the first finger. 


Term 2: 


e Play with fingers 1, 2 and 3 on the D string; Fiddle Time Starters exercises 38, 41, 43 and 46; 
e play with fingers 1, 2 and 3 on the A string: Fiddle Time Starters exercises: 52, 54 and 57; and 
e play with fingers 1, 2 and 3 on the E and G strings (violin), or C and G strings (cello). 


e Bow melodies with string crossing; 

e play harmonics; 

e read accidental symbols; and 

e play Fiddle Time Starters exercises 58, 69, and 70. 


Singing 


The singing curriculum was loosely based on the Kodaly Method as this was the singing method used in 
the study this trial replicates (Schellenberg, 2004). The Kodaly Method is a child-centred, developmental 
approach to music education developed in the mid-twentieth century. 


The core concept is that pupils are slowly introduced to musical skills, firstly through listening, singing or 
movement, and it is only once they are familiar with such concepts that they learn musical notation. 
Activities to reinforce the concepts include things such as games, movement, songs and exercises. The 
singing curriculum was split into two-week workshops (six workshops per term). For each workshop, a 
learning objective and an assessment criteria were given in the curriculum. For example, during workshop 
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one, the learning objective was to ‘show and feel the beat’, and the assessment criterion was to 
‘experience and learn about the pulse in music and to speak rhythmically and show the beat in various 
ways’. Tutors were advised to focus on the learning objectives of each workshop but to be flexible in their 
approach when teaching. 


For each lesson plan there was a suggested warm-up activity, several learning activities, and suggested 
vocal warm-ups. Tutors were also encouraged to review what had been achieved at the end of each 
lesson and to check learning objectives had been met. 


Learning objectives within the singing curriculum were as follows: 


Term 1: 


e show and feel the beat; 

e discover and use a singing voice; 

e demonstrate rhythm as a sub-division of the beat; 
e explore vocal range and tuning; 

e demonstrate musical memory; 

e problem solving and composing skills; 
e working with a partner; 

e distinguish pulse from rhythm; 

e perform a rhythmic ostinato; 

e recognise duration symbols; and 

e — sing in tune (most). 


e Show and feel the beat whilst singing/chanting; 

e use the voice in different ways (timbre); 

e consolidate accurate tuning on s, m and | (g, e, a); 

e develop awareness of beat and rhythm working together; 

e develop confidence in communicating own lyric ideas within a song, improvisation; 
e work creatively with timbre and lyrics; 

e sing in tune; and 

e differentiate between beat and rhythm. 


e To be aware of layers of sound (texture); 

e to differentiate between beat/pulse and rhythm; 

e link symbol to sound in rhythm work; 

e chant, sing and play in two parts with an increased vocal range; 
e explore rhythm through improvisation; and 

e understand the concept of rests as active in music. 


Drama 


The drama sessions were split into separate ‘blocks’, each of which covered a different story (one per half 
term). There were six stories covered in total with the first blocks lasting for 6 weeks each and the final 
blocks both lasting for 4 weeks (See Appendix B). The schedule for Drama sessions can be seen below: 
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Block 1 6 workshops: The Crashed Spaceship 
Block 2 6 workshops: The Orchid Hunters 
Block 3 6 workshops: The Snow Queen 

Block 4 6 workshops: Women of the Seas 
Block 5 4 workshops: Daedalus and Icarus 
Block 6 4 workshops: The Giant’s New Coat 
Block 7 2 reserve weeks 


The curriculum was based on Process drama (inspired by Dorothy Heathcote)* with some influences from 
Vivian Gussin Paley’s ‘helicopter technique’.° The curriculum aimed to incorporate extra-curricular 
learning into all stories, and to be interesting and educationally relevant to all pupils. The curriculum 
suggested that tutors needed to do background research and reading in order to confidently explore the 
story in each block. The delivery team suggested sources of additional information to supplement the 
curriculum. 


The drama curriculum was designed to: 


e build pupils’ confidence; 

e build pupils’ communication skills; 

e encourage pupils to focus; and 

e encourage imagination and thinking. 


Activities within the lesson plans included a mixture of improvisation, tutor narration, discussion and 
physical activities such as ‘pass the invisible ball’. 


Tutors were encouraged to be flexible and use their professional experience to adapt activities to different 
ability ranges and confidence levels, for example by simplifying or reducing the number of activities in a 
workshop if necessary. If one exercise was either proving very beneficial or was not going well within the 
session, tutors were encouraged to be flexible with the lesson plans as long as the overall story was being 
covered within the block. 


Curriculum flexibility 


As outlined above, the curriculum for all three arts groups provided suggested activities, learning 
objectives (or termly milestones), and suggested content (music to learn within the singing and strings 
sessions and stories within drama). However, there was a strong emphasis on flexibility and freedom for 
the tutor in terms of delivery style and technique, with the ethos that as long as the learning objectives 
were worked towards, tutors should be able use their own professional judgement and experience to 
achieve them. This meant that tutors chose different techniques, materials and activities within each 
session. There was also variation in the order, and pace in delivery, of the suggested content. These 
factors are very likely to have led to variation of delivery between tutors. This is further discussed within 
the process evaluation section. 


Outcomes 


This evaluation was designed to test whether pupils who received a music intervention (strings or singing) 
made more progress in maths and literacy than pupils in the ‘active’ control group (drama). Therefore, the 
two primary outcomes used in the analysis were maths and literacy attainment. The specific outcome 


“Drama as a learning medium’ (http://eric.ed.gov/?id=ED130362), or Dorothy Heathcote on 
‘Education and Drama: Essential writings’, (Cecily O'Neill, 2014). 

° ‘The Boy Who Would be a Helicopter: Uses of Storytelling in the Classroom’, by Vivian Gussin 
(Paley, 1991). 
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measures were agreed in consultation with the EEF and the Creative Futures delivery team, with the 
independent evaluators leading discussions. The two primary outcome measures were: 


1. post-test standardised maths PIPS score; and 
2. post-test standardised literacy PIPS score. 


The paper-based Performance Indicators in Primary Schools (PIPS) assessments were used to measure 
pre- and post-intervention maths and literacy ability. PIPS educational assessment has been development 
by the Centre for Evaluation and Monitoring (CEM) at Durham University and is used in schools 
worldwide. PIPS has been shown to have high reliability using test and re-test at the beginning and end of 
the school year, as well as an accurate predictor of later educational achievement.® 


Administration of PIPS tests 


Year 1 pupils in participating schools who would be eligible to take part in the Act, Sing, Play programme 
when they moved into Year 2, undertook the pre-test between June and July 2013. A small number of 
pupils completed their pre-tests in September 2013 if they were new to the school, or unable to be tested 
for another reason during the summer term. Pupils were divided into groups of 5 to 6 to complete paper 
PIPS maths and literacy tests at the pre-test stage. Each test took approximately 30 minutes to complete 
(including instruction time) with each group of pupils. Tests took place during the school day and were 
administered by NatCen interviewers who supervised the completion process. All interviewers were 
subject to full CRB/DBS checks and received project-specific training prior to administering the tests. 
They were also responsible for making appointments with participating schools and collecting 
administrative (demographic) data from those schools. Pre-intervention PIPS tests were conducted with 
920 primary school pupils in total. 


Completed tests were returned to the CEM for marking, and the scores uploaded to secure internet files. 
The datasets were then downloaded from the PIPS+ website and converted to SPSS files for analysis by 
NatCen. 


Post-intervention maths and literacy tests were collected using the same procedure as employed for the 
pre-test. Once again, PIPS maths and literacy tests were administered in schools by NatCen interviewers 
during the school day and returned to CEM for scoring. Post intervention testing took place between June 
and July 2014. Interviewers and markers were blind to allocations throughout. 


Sample size 


Nineteen primary schools took part in the Creative Futures evaluation. Maths and literacy PIPS tests were 
completed by 920 pupils belonging to Years 1 and 2 within these schools. Pupils in the pre-test sample 
were randomised by statisticians at NatCen using stratified randomisation. This process began by 
removing those without a valid outcome code (that is, a code to indicate that their maths and literacy tests 
had been completed) and those deemed to be ineligible. The remaining 909 pupils were stratified within 
class by eligibility for Free School Meals (FSM), sex, and date of birth before being assigned to the 
intervention or control groups. All 909 of the pupils who were randomised were eligible for the post- 
intervention PIPS test. 


When designing the study a minimum detectable effect size of 0.2 was assumed.’ This is the smallest 
effect size that, if true, would yield an estimate of impact statistically significant at the 5% level, based on 
the following assumptions: 


® See http://www.cem.org/primary 
’ By effect size we refer to the standardised mean difference (Cohen’s d). 
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e — probability—statistical significance of 95% to detect effects with a probability of <O.05; 

e power—80% power; and 

e randomisation—a 1:1:1 ratio of pupils (stratified within class by FSM eligibility, sex, and date of 
birth). 


Provisional calculations suggested that it would be necessary to include approximately 850 cases in the 
final analysis sample (570 in comparisons of one intervention arm against the control arm). Estimates 
were based on a comparison of outcomes between the treatment groups and the ‘active’ control group. A 
loss to follow-up rate of 5% was assumed, and 909 cases were included at randomisation. In practice loss 
to follow-up was somewhat higher at around 9% (N = 84). The tables below illustrate the detectable effect 
sizes at randomisation (recruited sample) and at the analysis stage (analysis sample). The computation of 
both sets of effect size includes an empirical school-level ICC of 0.2. 


Table 1a: Minimum detectable effect sizes—recruited sample 


Comparison of one study Comparison of intervention 


arm with another groups 1 and 2 with the 


ed \V{emexe] alice) me] cole] 0) 


Total sample size 605 909 
Unadjusted 0.22 0.19 
0.20* 0.19 0.17 
0.40* 
0.60* 


* Proportion of residual variance explained by covariates. 
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Table 1b: Minimum detectable effect sizes—analysed sample 


Comparison of Comparison of Comparison of 
treatment arm 1 treatment arm 2 treatment arms 1 and 2 
Wiidamexe) alice) m-laan Miidamexeyaliae) are laan) idem datsmexe) alice) m-lann 


Total sample size 814 


Unadjusted 


0.14 0.12 


* Proportion of residual variance explained by covariates. 


Randomisation 


The randomisation of pupils for this study was undertaken by statisticians within NatCen Social Research 
(the independent evaluation team) in September 2013 (the first week of the new term). Constrained 
randomisation was conducted within each class. With average class sizes of approximately 30 pupils this 
resulted in three groups of around ten children. Within classes, pupils were ordered by whether or not 
they were eligible for Free School Meals (FSM), by sex, and finally by date of birth. Within each class, 
blocks of three pupils were created from this ordered list, and within each block one pupil was randomly 
allocated to each of the three treatments (using random numbers generated by SPSS). The overall 
allocations were then checked by FSM, age and gender, plus other variables including ethnicity, SEN and 
English as an additional language (EAL). Pupil characteristics data was initially provided by schools and 
supplemented with information from the National Pupil Database at the analysis stage. 


Analysis 


The impact analysis set out to test whether the pupils who received a music intervention (strings or 
singing) made more progress in maths and literacy than children in the control group (drama). The 
primary outcome measures were post-intervention standardised PIPS test scores in maths and literacy. 


Analysis was conducted in STATA version 13 (Stata Corporation, College Station, Texas, USA) and 
impacts estimated in accordance with the intention-to-treat (ITT) principle. This preserves the effect of the 
initial randomisation by providing conservative and unbiased estimates of the effect of the intervention on 
pupils, regardless of whether they went on to receive the intervention. All 909 pupils who were randomly 
allocated to the treatment or control groups were eligible for post-intervention testing. 


Initially, descriptive analysis was run to look only at pupils with primary outcome scores in maths and 
literacy. This used standardised PIPS scores to compare the intervention and control groups. 


Hierarchical linear regression modelling (known as multilevel modelling—MLM) was employed to compute 
the effect sizes on the two primary outcome measures using standardised scores which provide an 
indication of pupils’ ability in maths and literacy. This involved fitting a multi-level linear regression model 
with random intercepts; the pupil being level one in the model, and the school level two.® Multilevel 
modelling accounts for clustering at the school level. 


® Tests were performed to explore whether pupils were clustered by class. Multilevel models were 
implemented with class as a second level (along with fixed effects for schools). However, the decision 
was taken to exclude class from the second level when the analyses showed that pupils were not 
clustered by class. Both the ICC (intra-class correlation coefficient) and the second level variance 
term were practically equal to zero. This indicates that the class-level means at post-test do not differ 
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The following covariates were included in the adjusted analysis: 


e Pupil level: allocation (whether the pupil was assigned to the strings, singing or drama group),” 
pre-test PIPS scores in maths / literacy, FSM eligibility, EAL status, ethnic group, sex, and month 
of birth. 


e School level: no covariates were specified at level two. 


All models were re-estimated in a separate analysis including only those who qualified for free school 
meals. Interaction tests were also conducted in order to confirm findings produced by these models. *° 


Effect sizes and their respective 95% confidence intervals were calculated following the procedure set out 
in Tymms (2004): 


gs =f 
o 


Where £ represents the adjusted difference in outcomes between intervention and control groups 
obtained from the adjusted multi-level model, and othe square root of the pupil-level variance obtained 
from fitting an unadjusted multi-level model.’* The unadjusted model contains a constant, and an 
intervention dummy variable, but no further covariates. The standard error of the effect size was 
computed according to Schagen and Elliot (2004). Finally, summary descriptive statistics are produced 
along with estimates of intra-class correlation coefficients (ICCs) for each regression model estimated. 


Process evaluation methodology 


The process evaluation was designed to explore how the class teachers and arts group tutors viewed the 
delivery and implementation of the programme, and what they perceived the impact to be. A qualitative 
case-study approach was used to meet these objectives and involved in-depth interviews conducted in 
five schools. Each case study visit included observations of strings sessions and interviews with: 


e strings tutors; 

e — singing tutors; 

e drama tutors; and 

e class teachers. 

e Feedback was gathered from the perspectives of both the deliverer (the tutors) and the host 
school. 

e Observations of the string sessions took place first, followed by interviews with teachers and 
tutors. Observations focused on three themes (see Appendix D for proforma): 

e pupil engagement and behaviour within the session; 

e fidelity to the programme; and 


from the grand mean. This is theoretically justified as primary school pupils are not assigned to 
classes based on ability. 


: Dummy variables were created to indicate which of the intervention groups the pupil had been 
allocated to. The analysis was disaggregated into three sets: intervention 1 (strings) versus the 
control group (drama); intervention 2 (singing) versus the control group; and intervention 1 and 2 
versus (music) the control group. 


*° The interaction models were implemented using a random intercept and random slope hierarchical 
regression model (multilevel model). However, an interaction on level one (pupil level) was also 
specified between FSM eligibility and allocation (being in an intervention group versus the control 
group). This analysis indicates whether the effect of being in a treatment group or a control group (in 
other words, being exposed to music or drama) differs for FSM pupils as a sub-group. 


™ The pupil-level variance is the variance of the level 1 error term indicating how a pupil’s level on the 
post-test deviates from the mean level of the post-test for the school the pupil belongs to. 
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resources and equipment available. 


Sampling and recruitment 


In order to carry out case study visits in a diverse range of participating schools, the following information 
was used to select the case study sample: 


levels of FSM take-up amongst pupils; 

levels of EAL and SEN pupils within the participating year group; 

levels of disruptive behaviour in sessions (as reported by teachers); 

whether teachers had reported that they would like to continue with the programme; and 
whether or not pupils were encouraged to take their instruments home and practice. 


In-depth interviews 


The content of each interview was based on a topic guide to ensure systematic coverage of key issues. It 
was intended to be flexible and interactive, allowing issues of relevance for individual respondents to be 
covered through detailed follow-up questioning. For teachers, the interviews covered questions on: 


the profile of the school and background information about the teacher, including their role within 
the programme; 

the programme, and any recommendations they had about how it could be improved; 

the format of the lesson and any differences they could observe during the lesson between 
different subgroups; 

the pupils’ attitudes towards the programme, and any perceived impacts they felt the programme 
was having; and 

the barriers to the implementation of the programme and to sustaining it within their school 
beyond the research period. 


For the tutors, the interview covered questions on: 


the tutors’ background and experience of teaching; 

the nature and the effectiveness of the training sessions (including how prepared tutors felt to 
deliver the programme); 

whether the programme met its aims, and any recommendations they had about how it could be 
improved; 

the lesson itself—including the content each tutor covered within their lesson, and the support 
they received from others (Such as teachers and the programme team); and 

pupils’ attitudes towards the programme, and any perceived impacts tutors thought the 
programme was having on pupils. 


Interviews were digitally recorded and subsequently analysed using Framework, a systematic approach to 
qualitative data management developed by NatCen Social Research and now widely used in social policy 
research. All participants were told that everything discussed in the interview would remain confidential 
and would be treated in accordance with the Data Protection Act. Additionally, it was made clear, both in 
recruitment materials and during the interview, that teacher and tutor views or opinions would only be 
reported anonymously. 
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Impact evaluation 


Timeline 
Date Activity 
2014 


Participants 


School recruitment was undertaken by the Creative Futures delivery team from January to May 2013. The 
recruitment process differed across the four areas included in the study, although in all cases schools who 
offered music instrument teaching in Years 1 and 2 were excluded on the basis of eligibility criteria. 


In Coventry, schools were selected by the Coventry Performing Arts Service (Coventry's Music Education 
Hub). Staff at the Service were guided by their own knowledge about local schools and how likely they 
believed schools would be to engage with the intervention. Of the eight schools approached about 
participating in the project, six accepted, however one of the schools later withdrew from the study due to 
the scale of pupil testing. This school was replaced by an additional school based in Essex prior to pre- 
test and randomisation. 


In East Sussex, a letter was sent by the Creative Futures team, in conjunction with the Music Education 
Hub, to all primary schools in the area informing them about the intervention and inviting them to take 
part. Schools were selected from all those willing to participate on the basis of the core eligibility criteria 
and whether they had sufficient physical space to accommodate the project. * Schools with two-form 
entry were prioritised, although the East Sussex school group did include two single-form entry schools. 
Twenty-nine schools were approached in total in this area. Although four schools accepted the invitation 


* Three rooms were required so that the activities could take place simultaneously. 
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initially, one of the schools withdrew from the study due to insufficient flexibility in terms of timetabling. 
This school was also replaced by an additional school based in Essex prior to pre-test and randomisation. 


In Essex, eligible schools were assessed and selected by the Creative Futures team with assistance from 
the Essex Music Service, with the delivery team making initial contact with the schools. In this case, 
selected institutions included small, single-form entry schools based in rural locations and larger, two-form 
entry urban schools. The schools were clustered in certain geographical locations for ease of delivery. Of 
the ten schools in this area approached to take part in the study, seven agreed to participate in the 
programme. One additional school was held in reserve, but was not included in the study. 


In Hounslow, four schools were selected by the Hounslow Music Service based on the core eligibility 
criteria. All four schools approached to take part in the study accepted the offer and became part of the 
intervention. 


All schools invited to take part in the study were sent an information pack and a school agreement form 
(the Memorandum of Understanding) by the Creative Futures team. Headteachers were asked to return 
the signed agreement to the Creative Futures team to confirm their place in the intervention. 


A total of 51 schools were approached to consider taking part in the intervention, 30 of these schools 
declined to participate in the study. Two schools withdrew prior to the pre-testing period, but were 
replaced by additional schools recruited in another area. In all, out of 51 schools approached, 19 were 
taken though into the final randomised sample. 


Pupil recruitment 


Opt-out letters were sent to parents of all children in participating classes. These were produced in 
collaboration with the delivery and independent evaluation teams, and distributed by the delivery team to 
all participating schools. Participating schools were asked to distribute letters to the parents of every child 
in classes included in the intervention. The letter explained to parents what the Creative Futures 
programme involved, and offered them the opportunity to opt out of the intervention. Parents were asked 
to let their school know if they did not wish their child to take part. The opt-out applied to the intervention 
and evaluation process, and not the separate components. By consenting to the intervention, parents also 
consented to the PIPS testing. No opt-outs were received from parents. 


Participant flow 


Figure 1 shows the flow of participants through the study. A total of 909 pupils were in the study at the 
point of randomisation.*® 


In September 2013, 909 pupils belonging to 33 classes in 19 schools were randomised—301 to the first 
intervention group (strings), 304 to the second intervention group (singing), and 304 to the control 
condition (drama). 


The final number of pupils who had post-test data and were eligible for unadjusted primary analysis was 
269 in the first intervention arm (strings), 277 in the second intervention arm (singing), and 279 in the 
control arm (drama); a total of 825 pupils. The adjusted analysis includes only those cases with a 
complete set of pre- and post-test data; this reduces the total number of pupils to 814: 269 in the first 
intervention arm, 272 in the second intervention arm, and 273 in the control group. This meant an attrition 
rate of approximately 10%. 


*S Eleven pupils were removed from the sample file prior to randomisation; ten of these pupils were 
missing pre-test scores, and one was removed on instruction from test administrators. 
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School characteristics 


Nineteen primary schools participated in the Act, Sing, Play Programme. The mean capacity of schools 
taking part in the intervention was 326 pupils (Table 2). The majority of schools were ‘community schools’, 
with 11% classed as ‘foundation schools’, 11% as ‘voluntary aided’ and 11% as ‘voluntary controlled’. The 
nature of the programme meant that only infant and primary schools were included in the study—the 
majority of these (95%) having an upper age limit of 11. In terms of FSM eligibility, the schools were 
slightly above average, at 19%, compared to an average of 17% of pupils in primary schools in England 
being eligible for and claiming FSM (DfE 2014). 


Table 2: School-level characteristics 


Characteristics 


Pupils as Mean number of pupils per school 
randomised (SD) 
Median number of pupils per school 
(Min; Max) 
eligible for FSM (SD) 
are eligible for FSM (Min; Max) 
Intervention—Strings group (SD) 
Median number of pupils in the 
Intervention—Strings group (Min; Max) 
Mean number of pupils in the 
Intervention—Singing group (SD) 
Median number of pupils in the 18 (6; 27) 
Intervention—Singing group (Min; —_— | 
Max) 
Mean number of pupils in the 
Control—Drama group (SD) 
- Drama group (Min; Max) 


Note: Percentages may not sum to 100% due to rounding. 
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Class characteristics 


A total of 33 classes were involved in the intervention across the 19 schools (see Table 3). Each class 
included an average of nine pupils assigned to each of the three arms of the intervention (ranging froma 
minimum of 6 toa maximum of 13). When looking only at FSM pupils, the number of classes varied a little 
across the intervention arms, with 20 classes with FSM pupils in the first intervention group (strings), 24 
with FSM pupils in the second intervention group (singing) and 22 classes with FSM pupils in the control 
group (drama). The number of FSM pupils in each class showed little variation across the different 
intervention arms. 


Table 3: Class-level characteristics (as randomised) 


Intervention Talc) avdevald(e) a) feXeyalige) Melts Koti 
class — Strings class - Singing Drama 


All pupils Number of pupils 301 | 304 304 
in trial 
Number of | 33 | 33 33 
classes 


Mean (SD) 9.1 (1.4) 9.2 (1.4) 


FSM pupils only | Number of pupils 61 62 
in trial 
Number of 20 24 
classes 
Mean (SD) 3.1(1.7) 2.6 (1.5) 
Median (Min, 3 (1; 6) 2 (1; 6) 
Max) 


Pupil characteristics 


Median (Min; 9 (7; 13) 9 (6; 13) 
Max) 


Tables 4 and 5 present a summary of pupils’ characteristics in control and intervention groups as 
randomised (Table 4) and as analysed (Table 5). At randomisation, 909 pupils from 19 schools were in 
range of the trial. Restricting analysis to just the pupils with post-test scores (as analysed) reduces the 
total number to 825 (269 and 277 in the two intervention arms, and 279 in the control group). 


The baseline characteristics of pupils across the intervention and control groups were similar at 
randomisation and again at analysis, suggesting that the loss of pupils between randomisation and 
analysis is unlikely to have introduced bias into the sample. There are slight differences in distributions of 
scores at Key Stage 1: for example, the singing intervention group shows a higher proportion of pupils 
with KS1 Reading L2b at randomisation, and the drama control group, with KS1 Reading L3 at 
randomisation and analysis (both reach conventional levels of statistical significance). 
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Table 4: Pupil characteristics as randomised 


All pupils in the 19 participating schools 


(as randomised) 


Intervention —_ Intervention (Oxo) alice) Difference Difference 

1 (strings) va (ilate plate) (drama) ((@fo) a=1ae= ys D) en ( Xo) a1 =1a sm DD) 

Strings and Singing and 
fol elants! drama 


45.6 46.8 0.1 “1,2 

(0.02) (-0.14) 

49.3 48.2 48.9 0.4 -0.7 
(0.04) (-0.09) 

10.0% 11.8% 10.2% 
20.3% 19.4% 24.7% 
32.0% 30.3% 29.3% 


KS1 21.7% 30.9% 21.7% 
Reading: 
L2b* 


Reading: L2c 
Reading: L3* 


0.0 
: 
KS1 2.3% 2.0% 1.6% 0.7 
Reading: 
Female 47.3% 49.0% 45.7% 


Mean pre- 
test score: 
maths 
Mean pre- 
test score: 
literacy 
KS1 Maths: 
L1 

KS1 Maths: 
L2a 

KS1 Maths: 
L2b 

KS1 Maths: 
L2c 

KS1 Maths: 
L3 

KS1 Maths: 
Missing 
KS1 
Reading: L1 
KS1 
Reading: L2a 


9 


1.6 
-5.3 
0.7 
-1.0 
1.0 
.2* 
Missing 
: : 3.3 
: : 0.7 
0.7 


Asian 17.7% 17.8% 17.1% 


Chinese 0.3% 0.7% 0.3% 
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Number of 
pupils (all) 
Note:* p < 0.05 


Table 5: Pupil characteristics as analysed 
All pupils in the 19 participating schools 
(as analysed) 


Intervention 1 Intervention —_— Control puierenee Dietenee 
(Guaiaes) PAiaelias)) (drama) | (Cohen’sD) — (Cohen's D) 
Strings STraleliarepeclare| 
Flare mele-lirs) drama 
Mean pre- i 45.7 
test score: 
maths 
Mean pre- 49.4 48.2 
test score: 
literacy 
KS1 Maths: 6.3% 3.2% 4.3% 2.0 -1.1 
L1 
KS1 Maths: 36.8% 33.6% 37.3% -0.5 -3.7 
L2a 
KS1 Maths: 27.1% 30.7% 23.7% 3.4 
L2b 
KS1 Maths: 8.9% 12.3% 10.8% -1.9 
L2c 
KS1 Maths: | 20.4% | 19.9% | 23.7% -3.3 
L3 
KS1 Maths: 0.4% 0.4% 0.0% 0.4 
Missing 


Reading: L1 
Reading: 
L2a 
Reading: 
L2b 

Reading: 
L2c 


KS1 29.4% 24.9% 32.3% -2.9 
Reading: 

L3* pd 
KS1 0.4% 0.7% 0.4% 0.0 
Reading: 

Missing — 
Female 48.0% 50.2% 45.2% 2.8 

EAL 24.5% 32.1% 31.2% -6.7 


FSM 19.3% 19.5% 17.9% 1.4 
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Ever FSM 20.4% ae ae ea 
SEN 21.6% aoe ae 
iia ace Bea el 
Asian 17.1% oe ae ee 
pupils (all) 


Outcomes and analysis 


Initial analysis is based only on the post-test scores for the primary outcome measures (standardised 
maths and literacy PIPS scores), without taking into account pupils’ pre-test scores or characteristics. As 
shown in the summary statistics for the post-test score on the first primary outcome—the PIPS maths 
score (Table 6)—the standardised scores for the first and second intervention arms (strings, 50.1; singing, 
49.5) as well as from the control arm (drama, 50.4) are very similar. As analysed, the sample of pupils 
consists of 825 pupils in 19 schools. Post-test scores in maths were missing for approximately 11% of 
pupils in the first intervention group (strings), 9% of pupils in the second intervention arm (singing), and 
8% of pupils in the control arm (drama). 


Table 6: Unadjusted average scores—PIPS maths score 


Intervention 1 Talecavcqvalece)a wy (exe) aia ae)| Total 
(strings) ‘Tate Tiare) 


(ole Vait-)) 


Sample as n=279 n=825 
analysed pupils 


Primary outcome Standardised post-test PIPS maths score 


Primary outcome 50.1 (10.5) 49.5 (9.8) 50.4 (10.7) 50.0 (10.4) 
mean (SD) 

Median (Min; 49.5 (22.0; 78.6) 49.5 (24.3; 74.5) 50.8 (22.0; 78.6) 49.5 (22.0; 78.6) 
Max) 


Missing (% of 8.2% 9.2% 
those 
randomised) 
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Table 7: Unadjusted average scores—PIPS literacy score 
Intervention 1 Intervention 2 Oxo} ay ice) | ae) r:|| 


(strings) ‘Tate [iate)) (drama) 


Standardised post-test PIPS literacy score 
49.4 (8.0) 49.7 (8.5) 


50.1 (27.6; 71.3) | 50.1 (19.5; 78.3) 


Table 7 shows the results of the second primary outcome—the standardised literacy score. Again, the 
difference between the two intervention groups (strings, 50.1 and singing, 49.4) and the control group 
(drama, 49.7) is negligible. Approximately 11% of pupils in the strings intervention group are missing post- 
test literacy scores, compared to 9% of the singing intervention group and 8% of the drama control group. 


Sample as 
analysed pupils 


Primary outcome 


Primary outcome 
mean (SD) 


Median (Min; 
Max) 


50.1 (8.7) 


49.7 (8.4) 


50.7 (19.5; 74.0 | 49.8 (22.7; 78.3) 


Missing (% of 
those 
randomised) 


Impact of music tuition on PIPS maths scores 


Analysis of the primary outcome was undertaken using a multilevel regression model. Results of both the 
unadjusted and adjusted analyses are presented. The adjusted analysis included the baseline measures 
as covariates: these covariates were students’ pre-test PIPS scores (maths/literacy), FSM eligibility, EAL 
status, ethnic group, sex, and month of birth. 


Table 8 shows the difference between two of the allocated groups (strings and drama) on pupils’ post-test 
standardised maths scores. In both the adjusted and unadjusted analyses, differences were very small 
and did not reach conventional levels of statistical significance, indicating that there is no evidence that 
participating in the strings workshops (compared to the drama workshops) had an impact on pupils’ 
standardised maths scores. 


The adjusted analysis was conducted on a final sample of 542 pupils across 19 schools. It reveals a 
difference of -0.30 (95% Cl: -1.37 to 0.79) in the standardised PIPS maths score at post-test. This is 
equivalent to an effect size of -0.03 (95% Cl: -0.13 to 0.07). 


Table 8: Analysis of primary outcome, maths—intervention (strings) versus control (drama) 
Unadjusted analysis Adjusted analysis 


-0.03 
(-0.13 to 0.07) 
-0.18 -0.30 
(-1.80 to 1.44) (-1.37 to 0.79) 
0.155 0.221 
(0.051) (0.065) 


17.05 11.37 
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Effect Size (Cl) 


Regression 
coefficients (95% Cl) 


ICC (SE) 


Variance school 
level (SE) 
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(6.53) (4.23) 


Variance pupil level 92.69 39.98 
(SE) 


(5.73) (2.47) 


Total sample size 542 (19) 542 (19) 
(Schools) 


Note: all effect sizes are calculated as adjusted regression coefficient, divided by the unadjusted standard deviation of pupil- 
level outcomes (see methods section for more detail). 


The impact analysis was repeated for pupils in the second intervention group (singing) compared to the 
‘active’ control group (drama). Results are presented in Table 9. As with the first intervention group, the 
difference between pupils’ post-test maths scores in both the unadjusted and adjusted analyses was very 
small and did not reach conventional levels of statistical significance. Therefore, there was no evidence 
that singing tuition (when compared to tuition in drama) had an impact on pupils’ PIPS maths scores. 


The adjusted analysis was conducted on a final sample of 545 pupils across 19 schools. It reveals a 
difference of 0.34 (95% Cl: -0.72 to 1.40) in the standardised maths PIPS score at post-test. This is 
equivalent to an effect size of 0.04 (95% Cl: -0.08 to 0.16). 


Table 9: Analysis of primary outcome, maths—intervention (singing) versus control (drama) 
Unadjusted analysis Adjusted analysis 
Effect Size (Cl) 0.04 
(-0.08 to 0.16) 
Regression -0.98 0.34 
Soeiicrents: (2270) (-2.55 to 0.59) (-0.72 to 1.40) 


ICC (SE) 0.153 0.197 
(0.051) (0.061) 


Variance school 15.69 9.34 
IEVENISE) (6.07) (3.54) 
Variance pupil level 87.11 38.08 
(SE) 


(5.37) (2.35) 


Total sample size 545 (19) 545 (19) 
(schools) 


The third model considered pupils from the two intervention arms as a single group (music) for 
comparison against the control group (drama). Results of the third model are presented in Table 10. In 
this case, the adjusted analysis was conducted on a final sample of 814 pupils across 19 schools. It 
revealed a difference of 0.03 (95% CI: -0.87 to 0.95) in the standardised maths score at post-test. This is 
equivalent to an effect size of 0.003 (95% Cl: -0.097 to 0.103). 


Once again the difference between pupil’s post-test maths scores in both the unadjusted and adjusted 
analyses was very small and did not reach conventional levels of statistical significance. We can therefore 
conclude that there is no evidence that music tuition had had an impact on pupils’ PIPS maths scores 
compared to tuition in drama. 
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Table 10: Analysis of primary outcome, maths—music versus control (drama) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) 0.003 
tama 

Regression -0.57 0.03 
Copiicients (2s70C)) (-1.94 to 0.80) (-0.87 to 0.95) 
ICC (SE) 0.153 0.207 

(0.048) (0.060) 
Variance school 15.99 10.19 
eVENE) (5.83) (3.66) 
Variance pupil level 88.35 38.94 
(SE) 


(4.43) (1.95) 


Total sample size 814 (19) 814 (19) 
(schools) 


Impact of music tuition on PIPS literacy scores 


Analysis of the second primary outcome—standardised PIPS literacy score—was undertaken using the 
same multi-level regression model. Again, results of both the unadjusted and adjusted analysis are 
presented. 


Table 11 displays results of the first ‘literacy’ model: this compares the first intervention group (strings) 
with the control group (drama). The adjusted analysis, conducted on a final sample of 542 pupils, reveals 
a difference of 0.21 (95% Cl: -0.68 to 1.10) in the standardised post-test PIPS literacy score at post-test. 
This is equivalent to an effect size of -0.03 (95% Cl: -0.08 to 0.14). 


As with the post-test maths score, in both the adjusted and unadjusted analyses, differences in the 
standardised literacy score were very small and did not reach conventional levels of statistical 
significance. Therefore, results indicate that there is no impact from the strings intervention on pupils’ 
literacy scores when compared to the drama control group. 


Table 11: Analysis of primary outcome, literacy—intervention (strings) versus control (drama) 
Unadjusted analysis Adjusted analysis 


Effect Size (Cl) 0.03 
(-0.08 to 0.14) 


Regression 0.30 0.21 
icl (0) 
eoeiticlents (a0 21) (-1.07 to 1.68) (-0.68 to 1.10) 


0.664 0.645 
ICC (SE) 0.081 0.069 
(0.033) (0.032) 


Variance school 5.86 2.01 
level (SE) (2.57) (1.00) 
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Variance pupil level 66.33 27.26 
(4.09) (1.69) 


(SE) 


Total sample size 542 (19) 542 (19) 
(pupils) 


As with the post-test maths score, the impact analysis was repeated for pupils in the second intervention 
group (singing) and control group (drama). Results of both the unadjusted and adjusted analysis are 
presented in Table 12. The adjusted analysis was conducted on a final sample of 545 pupils and shows a 
difference of 0.22 (95% Cl: -0.71 to 1.14) in the standardised PIPS literacy score at post-test. This is 
equivalent to an effect size of 0.03 (95% Cl: -0.09 to 0.15). 


Once again, the difference between pupils’ post-test literacy scores in both the unadjusted and adjusted 
analyses was very small and did not reach conventional levels of statistical significance; therefore results 
showed no evidence of an impact on pupils’ PIPS literacy scores, compared to drama. 


Table 12: Analysis of primary outcome, literacy—intervention (singing) versus control (drama) 
Unadjusted analysis Adjusted analysis 
0.03 
(-0.09 to 0.15) 
0.22 
(-0.71 to 1.14) 


Effect Size (Cl) 


-0.43 
(-1.74 to 0.89) 


ICC (SE) 0.071 0.005 
Variance school 4.70 0.16 
evENISE) (2.15) (0.41) 


Variance pupil level 61.09 29.64 
(SE) (3.76) (1.83) 


Total sample size 545 (19) 545 (19) 
(pupils) 


The final ‘literacy’ model brought together the two intervention arms (strings and singing) into a single 
group (music) for comparison against the control group (drama). Results are presented in Table 13. As 
with the equivalent model for standardised maths scores, the adjusted analysis was conducted on a final 
sample of 814 pupils. Results showed a difference of 0.23 (95% Cl: -0.55 to 1.01) in the standardised 
literacy score at post-test. This is equivalent to an effect size of 0.03 (95% Cl: -0.07 to 0.13). 


Regression 
coefficients (95% Cl) 


As in the previous models, the difference between pupil’s post-test literacy scores in both the unadjusted 
and adjusted analyses was very small and did not reach conventional levels of statistical significance. We 
can therefore conclude that there is no evidence that music tuition had an impact on either pupils’ 
standardised literacy or maths scores. There is no evidence of an impact of strings or singing tuition on 
pupils’ PIPS scores. 
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Table 13: Analysis of primary outcome, literacy—music versus control (drama) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) 0.03 
nary 

Regression -0.06 0.23 
coeiiiclents (2576e (-1.22 to 1.10) (-0.55 to 1.01) 
ICC (SE) 0.075 0.026 

(0.288) (.016) 
Variance school 5.15 0.76 
eVENE) (2.11) (0.45) 
Variance pupil level 63.35 28.04 
(SE) 


(3.17) (1.41) 


Total sample size 814 (19) 814 (19) 
(schools) 


Impact of the Act, Sing, Play intervention on subgroups 


Subgroup analyses were performed to examine whether there was evidence that the Act, Sing, Play 
intervention (strings or singing tuition) had an impact on the primary outcomes for FSM pupils as a sub- 
group (results are displayed in Appendix F). This analysis was carried out for both standardised PIPS 
maths scores and standardised PIPS literacy scores separately. Adjusted multilevel regression models 
were estimated for FSM pupils. 


Amongst the 100 pupils eligible for FSM there is no evidence of an impact of the strings intervention on 
the first primary outcome—standardised PIPS maths score: results reveal very little difference between 
allocated groups, equivalent to an effect size of 0.005. This did not reach conventional levels of statistical 
significance (95% Cl: -0.255 to 0.265). The equivalent result for the second primary outcome— 
standardised PIPs literacy score, which comprised 99 pupils eligible for FSM—was an effect size of -0.02 
(95% CI: -0.32 to 0.28) and again did not reach conventional levels of statistical significance. 


Similarly, no evidence was found that the singing intervention had an effect on either the first primary 
measure (standardised PIPS maths score), or the second primary measure (standardised PIPS literacy 
score), when compared to the control group (drama). Here the effect sizes were 0.01 (95% CI: -0.25 to 
0.27) and -0.03 (95% Cl: -0.31 to 0.25) for maths and literacy respectively. Neither result reached 
conventional levels of statistical significance. 


The third set of results shows no evidence of an impact of music tuition (strings or singing) on 
standardised maths PIPS scores, when compared to the control group (drama). The effect size was 0.02 
(CI: -0.20 to 0.24)—below conventional levels of statistical significance (n = 151). The second model, 
which looked at standardised literacy scores, did not converge. 


Full results from these analyses are presented in Appendix F (See Tables Fla to F6b). 
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Cost 


Cost information for the programme was provided by the delivery team and represents the costs of the 
programme as it was actually delivered—in this case, to a school with 2 classes in year 2, where each 
class was split into 3 groups (strings, singing and drama). The programme ran for 32 weeks. Creative 
Futures provided 3 days of monitoring and management. 


The costs are split between the ‘treatment’ (i.e. ASP-strings and ASP-singing) and ‘control’ (ASP-drama). 
Fixed costs that apply to both music and drama (e.g. curriculum design) have been split on a pro-rata 
basis. Costs assume that a school has a piano. 


Strings PTTaletiale) Music Total Drama 
(for music '(oeJ al dgo))) 


intervention) 


Number of students 40 20 


Tutor costs £2,560 £2,560 £5,120 £2,560 


Instruments £2,160 NA £2,160 NA 
(4x cellos; 16x violins) 


Curriculum design and resources £185 £185 £370 £185 
Strings resource books £160 £160 NA 


Management and monitoring £470 £470 £940 £470 
School total £5,535 £3,215 £8,750 £3,215 


Total per pupil, for year 1 £277 £161 £219 £161 


The total cost of the music elements of the programme was £8,750, which equated to £219 per pupil 
per year. 


A programme with similar aims to encourage more pupils to try music lessons could be delivered for 
lower cost to schools. Local Authorities have Music Education Hubs whose funding would allow them 
to offer the same or a similar programme without the same level of salary and instrument costs, 
however the exact cost would depend on the level of funding in particular areas, and the level of 
resource they have available. 


NO 
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Process evaluation 


In this chapter, findings from workshop observations and in-depth interviews are summarised in four 
themes: 


e implementation—covering training and school preparation for the Act, Sing, Play programme; 

e fidelity—how closely to intended the programme was delivered (focusing on classroom 
behaviour, pupil engagement, practicing and the curriculum); 

e perceived impacts—covering subject and soft skills; and 

e sustainability and future rollout. 


Implementation 
This section considers aspects of setting up the programme in schools. It covers: 


e training of tutors; 
e school preparation; and 
e lesson preparation. 


The main issue identified regarding implementation was that teachers and tutors believed that some 
tutors—particularly those with less experience of teaching primary children in schools—needed more 
guidance on how to run their sessions and how to manage pupil behaviour. This was predominantly an 
issue with strings tutors, who generally had less experience with teaching larger groups. Relatedly, it is 
worth emphasising that only strings sessions were observed, although in-depth interviews were 
conducted with strings, singing and drama tutors (as per the description in the methodology section). 


Tutor training 


The tutors for all three strands of the trial were either Creative Futures contractors (freelance tutors who 
were employed directly by Creative Futures), or tutors employed by their local Music Services and sub- 
contracted to Creative Futures for their time spent on this study. 


All tutors attended a three-hour training session prior to the start of delivery which was delivered by the 
Creative Futures at a venue closest to them (in Sussex, Coventry or London). This means that tutors who 
would be responsible for teaching the three different arts groups were all trained together. 


Tutors reported that the training was positive in the following ways: 


e It gave them a good understanding of what to expect when delivering the programme, and an 
understanding of its overall aims: 
‘It was good to get an idea of what to expect, what to look out for, how to get prepared for it, and 
know the content and exactly what we were aiming for.’ (Strings Tutor) 

e It provided them with advice about how to convey information to younger pupils. 

e _Itsuggested a ‘hands-on’ approach whereby tutors could take forward the curriculum in their own 
way. 

e —_Itinitiated peer support which provided tutors with an ongoing sense of support in addition to that 
provided by the trainers (such as sharing curriculum tips and best practice). 


However, the session was also seen as more of a ‘strategy’ session than formal training about how the 
programme should be delivered. Trainers discussed flexible ways that the curriculum could be 
approached rather than a prescribed way to deliver the sessions. The flexible approach was daunting for 
less experienced tutors: they would have preferred more guidance on exactly what to deliver and how. It 
was also too short, with insufficient detail on strategies for dealing with classroom behaviour. These 
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issues were considered more problematic for tutors with less experience of school-based teaching or 
experience of working with primary-aged pupils. 


Some class teachers reported that behavioural issues occurred in the Act, Sing, Play lessons and 
attributed this to the lack of training tutors had received. Those tutors who described themselves as less 
experienced in classroom teaching echoed this view and felt they were insufficiently equipped to deal with 
behaviour issues and managing a class of primary pupils despite some feeling confident and experienced 
in one to one instrument teaching. 


Tutors suggested that two aspects of the training in particular, would benefit from more development: 


e teaching the techniques required for the later parts of the strings curriculum; and 
e training in behaviour management. 


School preparation 


The impacts on the school during the time of the programme were minimal. For some teachers, having 
tutors come in and teach during one of their lessons meant that they gained time for other tasks. For 
others, however, the behaviour management issues experienced by tutors meant that they needed to sit 
in on sessions or help to manage behaviour during sessions. 


In some schools, changes to the daily routine needed to be made in order to allow for the programme to 
take place. The programme team worked with individual schools to find a suitable time and room for 
lessons to take place. This meant that pupils missed different lessons in different schools. For example, in 
one school the Act, Sing, Play session was held at the end of the day which meant that they moved their 
whole school assembly to the morning instead. It was considered much easier to adjust the timetabling in 
smaller schools as fewer pupils and classes needed to be considered. 


Teachers generally thought that the programme fitted well into their school structure, and that additional 
work for school staff was minimal. They did not report that timetabling the programme was an issue 
despite needing to arrange times for the sessions. Where additional work was required this included: 


e dealing with queries from parents about the programme (in particular, parents were keen that 
their child had the opportunity to learn an instrument and, during the trial, this was not possible for 
all pupils); 

e supporting and advising tutors about behaviour management, or discussing individual pupil's 
progress with tutors; 

e finding suitable storage for the instruments; and 

e investigating the possibility of pupils taking their instruments home to practice and, where this 
was not possible, finding solutions to allow pupils to practice in school. 


Some class teachers would have liked more information about the Act, Sing, Play curriculum, so that they 
could be more informed about the content of the sessions. Also, this would mean that if a tutor was 
absent they would be able to continue the session where possible. 


Having enough space was cited as an important factor for the effective delivery of the singing and strings 
sessions, however finding suitable space for these sessions was generally not an issue. Sessions 
generally took place simultaneously and in three different spaces within the school, such as the school 
hall and in the pupils’ main classroom. An additional consideration was space to store the musical 
instruments which was a challenge for some schools. 


There were mixed views from music tutors about the importance of having a piano available in the 
allocated room: some tutors considered this to be very important so that pupils could be accompanied 
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whilst playing or singing; others felt this was not necessary for a successful session as pupils could sing 
unaccompanied. During the trial, pianos were already available in the schools for tutors who required 
them. 


Violin rests (provided along with the instruments) were also seen as an important resource by strings 
tutors who felt that they helped pupils learn to hold the violin in the correct manner and made the activity 
less tiring for pupils. This meant that pupils could focus more on playing, rather than holding their violin. 


Session preparation 


Due to the flexibility of the curriculum and delivery of the programme, tutors’ preparation for sessions 
varied in both length and style. Some tutors planned for each session, drafting session plans and creating 
bespoke materials for pupils to use; other tutors planned just one or two activities for the session as they 
felt that pupils would struggle to engage with, and retain, more information. This difference in preparation 
style also meant that the length of time tutors spent preparing varied from less than an hour to two hours 
for each session. Tutors reported that they were happy to create extra resources and materials as they 
felt this was part of tailoring their own curriculum. They did not think that more resources needed to be 
provided by the Act, Sing, Play delivery team. 


Fidelity 


This section looks at how the sessions were delivered in schools in relation to what was intended. As 
detailed above, the programme was not set out in particular detail in terms of what should be covered in 
each session. This section, therefore, describes how delivery varied, and the reasons for this, under four 
themes based on workshop observations and interviews with teachers and tutors: 


e classroom behaviour; 

e pupil engagement; 

e §=curriculum content; and 

e instruments and practicing. 


Classroom behaviour 


The most common cause for loss of fidelity and progress within the arts Act, Sing, Play curriculum was 
the lack of behaviour control in the classroom. Some class teachers highlighted how unprepared tutors 
were for this, and how much of an issue it became for them in sessions. There were cases where tutors 
themselves thought that the lack of behaviour management was the biggest barrier for them in 
successfully implementing the programme. 


The types of behaviour issues which were encountered included: 


e pupils being over-excited; 

e pupils not staying on task; 

e pupils not listening; and 

e SEN and shyer pupils not engaging in the lesson. 


In some cases, school teachers recommended behaviour management techniques and advice to tutors, 
or attended the Act, Sing, Play sessions in order to help manage behaviour. Teachers felt that their 
presence should not have been necessary, and that the tutors coming into schools to provide tuition 
should have been prepared for this. In some cases, teachers attended sessions just for the first term until 
pupils got used to a new tutor. 


Some teachers recognised that their class was difficult to manage and that any new tutor would struggle 
to manage them. However, in other schools teachers felt that the inability of the tutors to manage 
classroom behaviour was due to their lack of experience and training with pupils of this age: 
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‘...[one tutor] particularly...he has asked before in the past, for me to go in and to sit in there 
because he said he can't control them when they’re off-task. But actually, they are the best 
behaved class in the school, to be honest, and they aren't difficult to control at all, they're just not, 
but he’s found it difficult. It’s to do with the way he presents his activities and getting them over- 
hyped. So I've had to go in... [Tutors] need to know more about very young children...teaching 
young children isn't quite as easy as some people might think. It is a skill and there are 
differences of approach’ (class teacher). 


Tutors whose previous experience of teaching was only in one to one sessions felt that the programme 
was a ‘learning curve’ for them in terms of developing behaviour management techniques. They described 
the pupils taking part in the programme as more ‘wild’ than those encountered previously. The issues with 
pupils misbehaving or losing engagement in the sessions affected the whole class and prevented tutors 
from delivering all of the content they needed to: 


‘There is one very naughty [child]: | tend to tell them just to be quiet and listen. | find the ones 
who are very naughty need more attention because of problems at school or at home, and it’s 
hard because | feel | give all my attention to them rather than the good ones’ (Strings tutor). 


Tutors cited three main sources of support when dealing with behaviour management and support with 
the programme more generally: 


e Classroom teachers provided tips and advice about individual pupils or whole class behaviour. 
They were aware of the school context, had knowledge of their students, and were the main 
source of support for tutors. Teaching Assistants were a valuable help within the Act, Sing, Play 
sessions, especially if tutors lacked confidence in managing classroom behaviour, however 
teachers saw this as unnecessary for working with small groups. 

e Support provided by peers and the delivery team was also valued by tutors: this was mostly 
through email communication. 


Behaviour was generally thought to improve over time with most pupils ‘settling down’ and getting used to 
the tutors and to the Act, Sing, Play programme as it progressed. 


Not all tutors felt that behavioural issues interrupted the flow of the programme—that with ground rules 
and clear expectations quickly outlined, behaviour was manageable. However, this view was at times 
contrary to that held by form teachers within the same school. 


Pupil engagement 


Generally, teachers and Act, Sing, Play tutors agreed that pupils enjoyed taking part in all of the sessions, 
and were excited to be taking part in a new activity. 


[Pupils] are excited and eager to get on and look forward to [their lesson].’ (Form Teacher) 


There were a number of reasons suggested as to what kept pupils engaged with, and learning from, the 
Act, Sing, Play sessions. These included: 


e the novelty of having a new tutor who did not normally teach them; 

e doing something ‘active’ and ‘different’ at school that was not academically based yet occurred 
weekly within structured sessions; 

e small group teaching helped with managing behaviour and setting up instruments, so reduced the 
amount of disruption and kept pupils engaged—having a small group of between five and eight 
pupils meant that tutors were more easily able to keep all pupils engaged; 

e having instruments that were new and their own made pupils excited to learn to play— 
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‘They enjoyed having something new and shiny... they liked it from the beginning, it was totally 
new, they just wanted to pick up and play their instruments’ (Form teacher); and 

e certain topics covered in the drama curriculum such as the ‘snow queen’ topic and the ‘ship’ topic 
particularly engaged pupils. 


Conversely, there were reasons why pupils were disengaged with sessions, no doubt reducing the 
amount of learning they were exposed to—and the overall impact of the programme: 


e Tutors struggled to manage behaviour effectively: 


‘| think that sometimes the [tutors] who’ve come in have not had the best behaviour management 
skills and that, especially at the beginning, sort of impacted on how much the children were 
Joining in. For example, | know that [one tutor] at first would say ‘oh you can choose whether to 
join in’, so they had a lot of children who would just wander round the room’ (form teacher). 

e Tuning, setting up string instruments, and giving practical individual support reduced the amount 
of ‘teaching time’ in each session and distracted pupils: 
‘...for example, if | teach bow hold...! have to come off teaching half way to go to each student to 
correct their bow hold, and when | do that with one student, the other one gets distracted...’ 
(strings tutor). 


In addition, some of the curriculum content was considered to be too advanced for the age group and not 
tailored to different ability levels. Strings tutors in particular found teaching different ability-level pupils 
difficult to manage, which led to some pupils disengaging with the programme. 


Programme and Curriculum 


Tutors were encouraged to be flexible in how they chose to meet the learning objectives provided by the 
delivery team. This meant that the suggested activities were not always followed, or were used in a 
different order or pace to that recommended by the delivery team. This led to varying levels of fidelity to 
the prescribed programme. 


Singing sessions 


The singing curriculum provided tutors with a number of suggested activities and objectives, however 
tutors were encouraged to add their own ideas, games and songs in order to reinforce the learning. The 
curriculum stated that tutors should focus on achieving the learning objectives, but that how they achieved 
these was up to them. 


Singing tutors were generally positive about the curriculum. They felt that it included enough detail to 
guide them, yet was flexible. This flexibility was valued as it enabled them to alter the content of their 
sessions, such as the songs they taught, to what would work better with a particular group or school. 


‘The kids absolutely love the songs they do and are so excited for singing lessons’ (singing tutor). 


Having a flexible curriculum also meant that tutors could vary the pace of the sessions. This was valued 
as differences in pace were needed for different classes of pupils. 


Whilst the delivery team initially provided a pack of activity resources, tutors were responsible for 
designing the majority of individual session plans. Tutors provided most of the session materials 
themselves (although a bank of activities were provided by the programme team initially) and so 
tutors were providing these as well as creating individual session plans. 


Strings sessions 
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The strings curriculum was designed in order to provide tutors with the freedom to use their own learning 
approach. However, for consistency the curriculum referenced Fiddle Time Starters, a beginner’s violin 
book, and also provided tutors with termly milestones and some suggested exercise/activity types. 


The strings curriculum was seen by some tutors as too ambitious for children of this age. This meant that 
in some cases pupils were struggling to meet the expectations set out in the curriculum. Of those who 
mentioned this issue, there was not a consensus about whether this was the case for lower ability pupils 
only, or for all pupils. 


The curriculum was too fast paced and was ‘too much to take in’ for pupils. This feeling was exacerbated 
by the fact that pupils were mostly not able to take their instruments home to practice. Tutors felt this 
meant that progress week-to-week was slower than anticipated. In addition, it was felt that the technical 
expectations of pupils were too high. 


Much like the singing tutors, strings tutors also valued the flexibility offered in the curriculum as this 
allowed them to slow down the pace of the sessions, focus on certain aspects of the curriculum in more 
detail, or change the content if it failed to engage pupils. For instance, one tutor focused on pupils 
playing their instruments within the sessions rather than learning to read music, swapping crotchets and 
quavers for numbers to simplify the task. 


There were a number of different barriers to effective delivery cited by strings tutors: 


e The start of the session was quite chaotic, as the instruments needed to be set up and tuned. 
Some form tutors were helping with this task to try and minimise the disruption. 

e ‘Something that’s worth considering is the time it takes to tune nine instruments’ (form teacher). 

e Some tutors felt that they needed two teachers within the classrooms to help them physically 
manage getting around the classroom to help each pupil with their instrument, to help with 
behaviour management, and to keep pupils on-task. 

e Some tutors would have liked more lead-in time to the sessions, including information about who 
the pupils were and whether they had any special educational needs or language difficulties. It 
was also felt that the string sessions, more than other sessions, were outside of the context of 
everyday life and so pupils needed more time to be introduced to the instrument and to the format 
of the sessions. 

e {The violin] is such a hard thing to learn, it’s so different and not in every family context. It would 
have been nice to have more time to introduce it to them, and more time to play around with it, 
instead of listening and doing what the teacher says all the time. | think they’ve missed out on 
that bit’ (strings tutor). 

e = The lack of pupil practice was cited as a major barrier to progress within the sessions, however 
some tutors thought this was less important for this age range and that one session per week 
was adequate. 


Drama sessions 


As with the strings and singing curricula, tutors were encouraged to be flexible and use their professional 
experience to adapt activities to different pupil ability ranges and confidence levels within the Drama 
curriculum. Lesson plans included suggested activities which were a mixture of improvisation, tutor 
narration, discussion and physical activities. 


Drama tutors were positive about the curriculum. In particular they felt that the first block worked well, 
however some tutors felt that some lesson plans were too fast paced with too many activities to fit in, 
and this, at times, made the sessions feel rushed and over full. 


Some of the content (two or three blocks in particular) was very popular with pupils and appropriate 
for use with them. However, certain aspects were felt to be too advanced for pupils of this age. In 
particular, it was felt that pupils needed to have a broader understanding of some of the topics before they 
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were able to dramatize them. One example given was that pupils were asked to consider whether they 
should remove a plant from the Amazon Rainforest that held medicinal properties at the risk of harming 
the indigenous people who lived there. It was felt that topics such as these were too advanced for pupils 
in Year 2. More generally, tutors felt that more story-based work, rather than that based on issues, worked 
better with young pupils. 


In contrast to the singing and strings sessions, drama tutors mentioned behavioural issues less 
frequently. Where these were mentioned, the issue tended to be that some pupils became too excited 
during the sessions (particularly when split into groups). 


There were a couple of barriers to successful implementation mentioned by drama tutors: 


e The pupils would have benefitted from a better introduction to the sessions, to the format, and to 
the new tutors. This would help (particularly shy) pupils feel more confident expressing ideas and 
sharing work. 

e School staff were not always aware of the aims of the drama group, and some tutors felt their 
presence was a ‘bit of an imposition’ to school staff. The full endorsement of the programme by 
all school staff was needed. Clearer information about the project aims and the tutors’ roles would 
have helped with this. 


Instruments and practicing 


Whether pupils were allowed to take violins and cellos home with them to practice was decided by 
individual schools, and so varied across participants. 


Reasons for encouraging pupils to take home their instruments included: 


e To allow them to practice. Although some teachers felt that most pupils were realistically not 
practicing routinely, taking home their instruments gave them this opportunity. 

e To overcome the lack of space to store the instruments in school. Due to lack of space 
elsewhere in the school, instruments were being stored in unsuitable places, including the 
disabled toilets, so it made more sense for pupils to take them home. 


Pupils in most schools were not allowed to take instruments home with them due to concerns about care 
of the instrument, pupils sharing one instrument, and school insurance policies not covering instruments 
off-site. 


Tutors—and some teachers—felt that opportunity to practice was a priority, so, if it was impossible for 
pupils to take instruments home, a few schools tried to incorporate practice time into the school day by for 
example, organising lunchtime clubs where pupils could practice informally. These sessions did not need 
to be run by music teachers; tutors thought that any practice would lead to quicker progress during 
sessions: 


‘Repetition is very important for that age, and doing it twice a week would make it more 
beneficial...not everyone does take [the instrument home], but the ones who have taken it home 
are the ones that have improved the most’ (strings tutor). 


Perceived Impacts 


There was an almost unanimous view that the programme gave participating pupils access to 
opportunities not normally available to them at this age. In particular, the opportunity to play an instrument 
was valued very highly by the tutors and school staff—particularly for pupils from more disadvantaged 
circumstances for whom the cost of lessons and instruments was normally prohibitive. Teachers saw the 


Education Endowment Foundation 39 


Creative Futures: Act, Sing, Play 


programme as offering opportunities to receive more specialist music tuition and a more specialist music 
curriculum for pupils. In particular, the opportunity to learn to read music was highly valued by teachers 
who felt that this was a skill these pupils would have no other chance to access. 


In terms of other impacts, teachers perceived the programme as having an impact on pupils in one of 
three ways: 


1. developing skills related to the programme as well as other areas of the curriculum— 


‘The class as a whole are better at clapping in time...! couldn’t know for sure but | think that things 
like being able to clap, being able to hear the beat is also going to help with things like hearing 
individual sounds in phonics [and] hearing the rhythm in a rhyme or in a poem’; 


2. developing skills related to the programme, but having no wider impact across other areas of the 
curriculum; or 


3. not developing any skills related to the programme or the wider curriculum: 


‘1can see the intention of it... | haven't got a musical background, but they don’t seem to have got 
very far with their violins...and the drama pupils seem to have a nice time [but] | just don’t see 
how it can [raise attainment]’ (form teacher). 


Teachers reported how they thought the programme could impact positively on pupils in other areas of the 
curriculum, including pupils’ core subject skills and soft skills. There were, however, very few cases in 
which they had actually seen this impact. 


The main benefits of the programme were thought to be: 


e Increase in pupils’ confidence. While some felt that this was just in the Act, Sing, Play sessions, 
others considered this to spill over into mainstream lessons. 

e Pupils were given the opportunity to use their creativity and exercise self-expression within a 
structured environment. Tutors considered this to be important as it taught pupils to use their 
creativity in constructive ways. 

e The music sessions helped in the development of pupils’ social skills. Some felt that the impact of 
this could be seen in mainstream lessons through increased peer-to-peer interaction, and 
improved listening skills of pupils. 

e As aresult of practicing listening skills (particularly during singing sessions), pupils were more 
likely to listen to the teacher more carefully rather than rushing into a task in class. 

e The Act, Sing, Play programme taught pupils about the importance of perseverance with a task. 

e There were links made by both teachers and tutors between the exercises within the music 
sessions and core subjects such as maths and English. Such links included using addition to 
count the beats in a bar, and learning pitch and rhythm impacting positively on pupils’ poetry and 
phonics work. 


Music tutors thought that music generally, and the Act, Sing, Play programme specifically, was an activity 
which was ‘inclusive’ in the sense that all pupils—including those with social, emotional or behavioural 
issues, and those with lower abilities—could engage with the content and be included in the activities. 
Having inclusive sessions such as these were considered to be vital for such pupils, particularly within a 
context where creative activities are regarded as increasingly ‘sidelined in the curriculum’. 


Sustainability 


Teachers had different views about the sustainability of the programme. Some teachers described how 
the aspirations they held for continuing the programme were constrained by practical and financial 
considerations. The decisions about the sustainability of the programme were discussed by class 
teachers but the ultimate decision about further delivery of the Act, Sing, Play programme would be made 
at a school level by headteachers. 
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The main reason given for wanting to sustain the programme was that it gave pupils the opportunity to do 
something they would not otherwise have the chance to do. In some schools, teachers were already 
continuing some activities from the programme. For instance, in one school teachers were providing a 
lunchtime club so pupils could continue to play their instruments. 


Generally, teachers spoke about continuing the strings sessions. Some teachers explained how they 
would prefer pupils to have the opportunity to learn an instrument despite feeling that they would also 
benefit from drama and singing. Other teachers wanted pupils to be able to try all three arts activities. 


In order to continue with the programme, schools expressed a preference for some aspects of the 
programme to be reviewed or developed further. These suggestions included: 


employing tutors with experience of working with this age group, or to offer training to develop 
this; 

training tutors in how to manage classroom behaviour; and 

committing to, and making provision for, pupils being able to practice at home. 


Other considerations influencing decisions about continuing with the Act, Sing, Play programme included: 


Changes to the National Curriculum (from September 2014) would mean less time available 
within the school day for music lessons. 

Teachers were of the opinion that the cost of purchasing instruments and paying tutors would be 
a challenge should the programme be continued in their school. Some teachers described this as 
‘having the aspirations but maybe not the money’ to continue the programme. Related to this was 
a concern that the programme would need to be extended to cover Years 2 and 3 so that pupils 
did not have to stop lessons after Year 2. 

Some teachers had concerns about the motor skills and coordination of some pupils in their 
current year groups. This raised questions for them about whether all pupils would be able to join 
in with playing an instrument should the intervention be rolled out more widely. 


Formative findings 


The process evaluation findings suggest that the Act, Sing, Play programme may benefit further from 
refinements in the following areas: 


Better communication between both tutors and teachers prior to the start of the programme. A 

meeting or briefing would: 

o help teachers and other school staff to understand the aims and broad content of the 
programme, and help to bring all staff on-board; 

O provide more information about pupils, including any potential barriers to participation such 
as SEN or EAL so more targeted resources and activities could be developed; and 

o detail the curriculum for each session so teachers would know the content of the sessions 
and what stage the pupils were at. 


More lead-in time for pupils so they get used to their instrument before learning begins fully, or 
develop group cohesion so they feel comfortable ‘performing’ in the sessions. 

Facilitating practice sessions during the day at school. 

Review the level of training given about behaviour management. For example by teaching tutors 
specific behavioural techniques for controlling classes of this age range. 

Consider having two teachers deliver strings sessions to enable them to help pupils to tune their 
instruments, set up, and hold their instruments, and help deal with behaviour management. This 
would, in turn, make it easier to deliver the sessions. 

Review the level of the strings curriculum, including technical expectations of pupils, the pace of 
the programme [sessions?], and how to cater for higher and lower ability pupils. 

The flexibility in how to deliver the curriculum was highly valued by tutors. This flexibility should 
be retained as far as possible. 


Education Endowment Foundation 41 


Creative Futures: Act, Sing, Play 


Conclusion 


| Key Conclusions 


1. This evaluation provides no evidence that ASP-music workshops had a greater impact on 
maths or literacy attainment than ASP-drama workshops. 


2. Analysis of students receiving free school meals similarly found no evidence that ASP-music 
workshops had a greater impact on maths or literacy than ASP-drama workshops. 


3. The process evaluation suggested that some tutors—particularly those with less experience of 
teaching groups of primary school children—needed more guidance on how to run their 
sessions. 


4. Although not necessarily typical, there were related concerns that some strings workshops 
struggled to keep students focussed on learning music. 


5. Class teachers reported that confidence and social skills had improved for some pupils. 
Teachers also felt it was important that children from disadvantaged backgrounds had the 
opportunity to learn a new skill that they might otherwise not be able to access. 


Limitations 


The number of pupils included in the randomisation (n = 909) was above the required number set out in 
the sample size calculations (n = 850), however, in practice the rate of attrition was around 10% (n = 95), 
resulting ina sample size of approximately 814 (around 543 for each of the two study arm comparisons). 
The sample characteristics in terms of observable variables were similar for the randomised and the 
analysed sample, and assuming a robust randomisation process, the differences between the groups are 
due to chance. Furthermore, the level of attrition was similar in all three trial arms, which increases 
confidence that there is no bias. 


Interpretation 


The results of this randomised controlled trial do not provide any evidence that music tuition (either in 
strings or singing) relative to drama had an impact on attainment in maths or literacy. These results fail to 
corroborate Schellenberg’s evidence that music tuition causes increases in the IQ of young children 
(2004). 


The use of an active control group means that it is not possible to assess whether a music intervention 
would have had a significant impact relative to a standard (non-active) control. 


From the process evaluation results, the key challenge for the programme in the intervention schools— 
particularly in the strings sessions—was the behaviour of some pupils. Due to them being disengaged or 
overexcited in the sessions they became disruptive and this is likely to have impacted upon the learning of 
all pupils in the group. Not all tutors were able to manage this behaviour and so struggled to deliver the 
programme fully or in the depth they would have liked. 


Furthermore, the flexibility given to tutors within the curricula (particularly the strings and singing curricula) 
meant that their learning approach, and the content of their lessons, varied considerably. The extent of 
this remains unclear but means that fidelity to the programme content was difficult to measure. 


Future research and publications 


As indicated, there is a body of literature showing the positive effects of music tuition on academic 
attainment, particularly in literacy. Given the lack of evidence of impact identified by this evaluation, a 
priority for future research would be to explore the effectiveness of the programme against an inactive 
control group. 
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Appendix 1: Strings curriculum (Week 1 example) 


Education 


reative 
Endowment 
Foundation utures 


Act, Sing, Play: 


Investigating the impact of arts provision on 
achievement 


String Curriculum 


Introduction: We understand that there are as many different string teaching methods as there are 
tutors. With this in mind we have designed this planning document which provides the freedom for 
tutors to use their own approach but that which also has, for reasons of consistency for research 
purposes, three key features: 


- Reference to Fiddle Time Starters pieces, a copy of which will be provided for every 
student 

- Aset of termly milestones 

- Reference to some of the common exercise/activity types. Some weeks these activities 
are specified on the lesson plan, some weeks these are left to the tutor to complete. We 
have also provided a bank of activities as a starting point although tutors may wish to 
supplement these with other activities from Fiddle Time Starters as well as their own 
materials. 
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WEEK ACTIVITY FIDDLE TIME 
STARTERS EXERCISE 


1 Warm up 
@ Introduction: parts of the instrument, materials used. 


Posture 
@ How to hold the instrument in rest position and playing 
position. How to take out and put away. 


Pitch/singing 


Notation 
@ Introduce names of open strings 
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Appendix 2: Drama curriculum (Example weeks 3 and 4) 


Education 


Endowment ative 
Foundation utures 


Creative Futures (UK) Ltd and the Education Endowment Foundation 
Act Sing Play - Investigating the impact of arts provision on achievement 


Introduction to KS1 Year 2 Drama Programme of Study 


Type of Drama: 

Based on Dorothy Heathcote inspired Process drama with some influences from Vivian Gussin Paley’s 
helicopter technique. Emphasis is on being present in the drama and understanding situation, 
character journeys and emotions though lived experience, rather than preparing a pre-written 
entertainment piece for an audience. 


The principle is based on the idea that the pupils have ownership of the stories and are an integral 
part of the telling of them, and therefore are strongly motivated to communicate as clearly as 
possible. The aim of all the activities is to build the pupils’ confidence, communication skills and focus, 
as well as encouraging imagination and thinking. It is very important to set expectations clearly and 
make sure there are ground rules that are adhered to, as without this there is a danger that it can 
become a threatening and embarrassing experience for some pupils. 


The Stories 

The stories are planned with a strong structure, but the pupils often create the details; this process is 
modelled by the practitioner when narrating parts of the story or in role. Questioning pupils and 
asking them for contributions to the story happens as part of this process, so the story will never be 


taught exactly the same way. 


There is 4 strong incorporation of extra-curricular learning in all of the stories. The reason for this is 
not in order to justify the place of Drama on the curriculum, but because the drama needs to be about 
something for it to have purpose and dramatic tension. The subject matter was been chosen to 
(hopefully!) be of interest and educationa! relevance to the pupils and the practitioner needs to make 
sure they have done some background research and reading necessary to be able to confidently 
explore the topic. There are suggestions for where to look for information for each block or 
workshops. 


The importance of dramatic tension is key as this gives intention to all the drama and it means that 
you avoid the problem of a child being expected to improvise a scene, not knowing what to do and 
getting embarrassed. The framing of the stories is also very important as it means that the 
participants are in situations that place them in relation to the unfolding action. Sometimes they are 
very much present in the action, sometimes looking at it from a distance (either temporally, or 
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physically/emotionally). This variety is important to ensure that the intensity of the experience is not 


too overwhelming, but it also helps the pupils develop reflective thinking skills. 


Delivering the Workshops 

TIR -Teacher in Role 

Many exercises require the practitioner to be in role, but this doesn’t mean you have to be good at 
acting and it is nat something to worry about if you are not an experienced performer, it just needs 
preparation. Sometimes it is more effective to down-play the characters, not putting on funny voices 
etc. which can startle some pupils and possibly cause disruption. Explaining that you wil! be playing 
characters is important to establish at the start, and you often indicate when you are going to co it 
verbally and using a symbolic piece of costume (eg a scarf). 

Playing roles and narrating: 

| have indicated suggested things to say here in italics, none of it should be seen as lines that have to 
be learned, they are just a guide to help you end pupils improvise. | have included quite a bit of 
imaginative detail in places as an indication of how you can approach each role or bit of narration ina 
way that might engage the pupils, but it really is up to you how you do it. It is a good idea to focus on 
the aspects of the story or characters that capture your own imagination and make the scheme 
unique to you and appropriate to your class. 

Behaviour management/space: 

If you have a carpet space, pushing a few desks away should be sufficient but you will need more 
space than just the carpet. Using your normal classroom clapping signals is a good idea, but you wil! 
also need to establish some rules about Drama — making sure that we do not laugh or mock peaple 
when they are acting is the most important one, but keeping all the behaviour management linked to 
the story is very effective. For example, keeping them on task can e linked to the story, eg. you need 
to collect all the orchids quickly or it will be dark. Most of the activities have behaviour management in 
mind so they are all contained to a certain degree but they can be further simplified or restricted with 
children with more challenging behaviour. Choosing pupils who are focussing well first when doing a 
story square, or giving out roles is a good pasitive behaviour incentive. Pupils can always have roles 
taken away from them end be given different things to do if they are finding something too 
challenging. 

Simplification/being flexible: 

Depending on the ability range and confidence of your pupils (and any other factors that might affect 
how sessions run), you may need to simplify or even cut down on the number of activities you do with 
the class. For example, whole class improvisations can be turned into large frozen images, or small 
group frozen images. In a story square, you can ask pupils to make frozen images rather than role 
plays. A frozen image can then come to life when they are more confident. If one exercise is proving 
very beneficia| and engaging for all the pupils, then you can be flexible with the plans, as long as the 
overall story has been covered in the whole block. Similarly, if something is going really badly, it 
would be much better to adapt and change what you are doing than try to push on with something 
that isn’t working. 

Remember, you can always narrate parts of the story quickly yourself if you have not got time to for 
one of the activities. 
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w/s3&4 Objectives: To take part in a group improvisation, understand 
basics of photosynthesis and imaginatively explore the 
possibilities of the story. 

1) Start with a simple circle game like Chinese whispers. 


2) Teacher narrates: Zeepob2 makes another call to the Establish focus. 


mission contro! to update them on what she hos 
learned. Using the story square, choose a few pupils to 
take turns in making a radio call telling them what she 
has discovered from the last lesson. TIR as Mission 
contro|, you can reveal some things about her 


planet/story in your responses. 


What, you mean you are actually on thot tiny blue and 
green planet 3" nearest the sun in the nine-plianet 


spindle?... Well l never, We had no idea they had 
intelligent life forms. So what is this ‘speaking’?... Oh, 
communicating through oral sound-making ond visual 
symbols... Gosh, so they don’t use (you can reaffirm 
whatever method of communicating the pupils decided 
on earlier here, eg brainwave pulses) then? How 
strange... 


Teacher narrates: The humans invite the alien toa 
banquet as they are also a bit worried as they haven't 
seen her eat anything yet. What would be the best 
things to serve to show off earthly cuisine? Pupils to 
suggest ideas and other pupils to act out making them 
in pairs using story squere method. 


Whole class improvisation. We are all at the banquet — 
set out the physical parameters of where all the food is 
clearly. TIR as alien, looks very strangely at the food, 


picks it up and doesn’t know what to do with it. 


Discussion: How do you think the Alien would eat? 


Teacher narrates: The Alien tried to communicate this 
by picking up a plant and pointing at the sun anda 
watering can. Discussion: What do you think she 
meons? Take discussion further to discuss what plants 
need to live. What would it mean if we only needed 
woter ond sunlight to live? How would the world be 
different? 
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Recall the events in 
the story from 
previous lesson. 


Participate in 
whole group 
improvisation 


Use imagination 
and verbal 
describing skills to 
describe 
knowledge of plant 
and animal life 


Contribute to story 
square 


Participate in 
whole group 
improvisation 
Develop miming 
skills 


Use imagination 
and verbal 
describing skills to 
think about 
possibilities of 
alternative ways of 
existing. 


48 


Creative Futures: Act, Sing, Play 


Appendix 3: Singing curriculum (Example weeks 1 and 2) 


Please add your own ideas, games and songs as you wish to reinforce the learning. 
Anything work particularly well ? Please share with the other tutors via the email 
list. 

If there is anything in these plans you don’t understand, please contact me direct. 
Focus on the learning objectives — how you get there is up to you; the activities are a 
guide so don’t worry if you add things or miss bits out or deliver things in a different 
order. When a song/activity has a long list of learning attached, please take two or 
more lessons to cover the sequence. | suggest g and e might be a more effective 
tonal centre than A and F# in the two note songs. 


Term 1 6 workshops 


Overall aim/theme: To experience the beat or pulse in music; to begin to 
discriminate between beat and rhythm; to discover 


different voices, including the singing voice 
Weeks 1&2 


Learning Objectives: we |Assessment Criteria: what I’m looking for... 


re learning to... 


Ww/s1&2 how and feel the beat |Experience and learn about the pulse in music; 


Speak rhythmically and show the beat in various ways; 


Discover and use a 


inging voice Explore different voices — 
Sirening (running voice up and down entire range and 
lextending this range), 


speaking/whispering/singing/funny voices etc; 


Sing songs with limited note range using (s-m-l) 


Engage: have a variety of ice breaker/ warms up your sleeve. Start with a welcome song of your 
choice and/or a rhythmic greeting (e.g. tap, tap, say your name). If appropriate, try ‘A Monster 
came to visit you and he roared’ (to the tune of ‘The animals came in two by twa’) Try also ‘Owl 
warm up’ for vocal placement 


Learning (split the learning sequence over the two lessons, with plenty of chances to redo, 
refine, and consolidate): 


1. Rep No. 16: Pease Pudding Hot - T speaks it to pupils adding a clap on each crotchet rest; T 
asks pupils how many claps they heard (three!); T takes speaking part and pupils provide the 
claps; 

2. Replace claps with expressive sounds: a blowing sound after “hot”, a shivering sound after 
“cold” and a vomiting sound (“blah!”) after “old”; again T takes speaking part and pupils make 
the three sounds; 

3. Change jobs: pupils now take speaking part and T makes the sounds or divide class into 2 
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4. Change jobs again: split class into four groups - (1) first phrase; (2) second phrase; (3) third 
phrase; (4) sounds, and perform the chant again. 

5. Teach the children to sing the song, starting on d 

6. Stand in a space and pass a bean bag/picture etc around the room on the phrases. Extn: Able 
pupils may be able to pass the object exactly on the rest! 

7. Tap the beat (two hands on knees) whilst singing the song 

8. Now T speaks the chant and taps the rhythm while pupils provide three sounds 

9. Now T taps the rhythm without speaking and pupils provide the three sounds; 

10. Change jobs as above 

i bs I Now T performs the chant in the thinking voice, without tapping the rhythm but showing 
the pulse by tapping her/his foot on the floor and have pupils provide the three sounds; 

i. T asks pupils what they noticed: explain idea/concept of pulse and beat in music 

13. ‘Try using ‘traffic lights’ for singing voice (green), clapping and singing in your head (amber) 
and singing in your head — teacher not indicating pulse)- (red). NB — change the light on the 
phrase with enough time for pupils to process 


14. Pupils invent other ways of showing the pulse — play the game ‘clockwork soldiers’ 


Vocal warm up: ‘Have you got your...voice?’ Also — ‘Hippos and Mice’ 

1. Now teach Song 1 (I, |, me oh my). Pass around a ‘pie’ on the beat (beanbag — show the 
children how to do this first and hold up the flow of the song if a child passes too slowly — speed 
up if too quickly etc). When the song gets to ‘cherry’ the child who has the ‘pie’ speaks their 
idea for a new pie. NB ask children to think and decide what they will say before the song 
starts! Next, ask some children to try to sing their response. Pupils can suggest other kinds of 
pie - apple/strawberry/chicken/etc... 

2. Teach ‘Bounce high, bounce low’ ( ‘Songs for Y2 ’). Use a large ball; pupil pushes the ball across 

the circle as they sing a child’s name on the final two notes. Gently correct this to enable children 

to find accurate tuning 

Extra material Jack in the box’ (songs for Y2) 


Review: Review what has been achieved and celebrate/perform at the end of each lesson. 
Where do we/you need to go next? 
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Appendix 4: Pro forma used by researchers in observations 


of strings classes 


School Ref: 


Pupil behaviour: 


Pupil engagement: 


Fidelity to the programme: 


Classroom setting: 


Resources/equipment: 


Other observations: 
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Appendix 5: Consent letter for parents 


A Education ative 


Endowment 


Foundation tures 


NatCGen 


Social Research that works for society 


Leading education 
and social research 


Institute of Education 
University of London 


FREE music and drama tuition for Year 2 pupils next year 


“Act Sing Play: Investigating the impact of arts provision on achievement” 


Dear Parent, 


Your child’s school is going to be part of an exciting national project next year involving 
free weekly music or drama workshops for all Year 2 pupils. In addition to offering a great 
introduction to playing an instrument, singing or drama, the project will give us a chance 
to look at the impact of these activities on literacy and maths skills. 


Here are some key facts about the project: 


e All activities will take place during normal school hours 

@ Musical instruments will be provided free of charge 

e@ There is no charge for any aspect of the project: it is completely free 

@ Some literacy and maths tests will be completed by children at the end of this 
summer term as well as at the end of the project next summer — these will be short 
and will be done during the school day as normal 

e Pupils will be randomly allocated to one of the three activities: some will learn a 
string instrument, some will do singing, some will do drama 

e No pupils or schools will be named in any published reports about the project, or 
on the internet 

e All tutors delivering the programme are experienced, professional, CRB-checked 
individuals 

e The project is being run and managed by a charity called Creative Futures (UK) 
Limited, with funding from the Education Endowment Foundation. 


More information about the project is attached. If you want to discuss any aspect of the 
project, or you have concerns about your child taking part, please contact your school. 


Thank you 


Pdi Fight 


Julian Knight, Creative Futures (UK) Limited 
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Appendix 6: Subgroup analysis of primary outcomes 


Table F1a: Analysis of primary outcome, maths—intervention (strings) versus control (drama), 
FSM pupils only 


Unadjusted analysis Adjusted analysis 
0.005 
(-0.255 to 0.265) 
0.05 
(-2.42 to 2.51) 
0.971 


0.322 


Effect Size (Cl) 


2.01 
(-1.83 to 05.86) 
0.305 
0.131 


Regression 
coefficients (95% Cl) 


ICC (SE) 


(0.101) (.124) 
Variance school 13.81 14.35 
Variance pupil level 92.28 30.24 


(SE) (14.05) (4.58) 


Total sample size 100 (14) 100 (14) 
(schools) 


Table Fib: Analysis of primary outcome, maths—intervention (strings) versus control (drama), 
FSM interaction model 
Adjusted analysis 


Regression coefficient of the interaction between FSM -0.90 
i 0, 
status and experimental cell (95% Cl) (-1.91 to 3.71) 


P-value 0.529 


ICC (SE) 0.222 
(0.065) 
Variance school level (SE) 11.43 
(4.24) 
Variance pupil level (SE) 39.94 
(2.47) 
Total sample size (schools) 542 (19) 
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Table F2a: Analysis of primary outcome, literacy—intervention (strings) versus control (drama), 
FSM pupils only 


Unadjusted analysis Adjusted analysis 


-0.02 
Regression 1.74 -0.15 
coefficients (95% Cl) (-2.51 to 2.20) 


P-value 0.292 0.899 


Effect Size (Cl) 


ICC (SE) 0.018 0.001 
Variance school 1.25 0.04 
Variance pupil level 67.14 29.27 
Total sample size 100 (14) 
(schools) 


Table F2b: Analysis of primary outcome, literacy—intervention (strings) versus control (drama), 
FSM interaction model 


Adjusted analysis 
Regression coefficient of the interaction between FSM -0.54 
i 0, 
status and experimental cell (95% Cl) (-2.85 to 1.78) 


P-value 0.650 


ICC (SE) 0.069 
(0.033) 
Variance school level (SE) 2.02 
(1.01) 
Variance pupil level (SE) 27.24 
(1.68) 
Total sample size (schools) 542 (19) 
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Table F3a: Analysis of primary outcome, maths—intervention (singing) versus control (drama), 
FSM pupils only 


Unadjusted analysis Adjusted analysis 


0.01 
Regression -1.52 0.11 
coefficients (95% Cl) (-2.30 to 2.52) 


P-value 0.424 0.929 


Effect Size (Cl) 


ICC (SE) 0.000 0.056 
(0.000) (0.107) 
Variance school 0.00 1.89 
lEvENNSE) (0.00) (3.67) 
Variance pupil level 89.30 31.64 
ee) (12.69) (5.11) 


Total sample size 99 (15) 99 (15) 
(schools) 


Table F3b: Analysis of primary outcome, maths—intervention (singing) versus control (drama), 
FSM interaction model 


Adjusted analysis 
Regression coefficient of the interaction between FSM 0.91 
i 0, 
status and experimental cell (95% Cl) (-1.84 to 3.67) 


P-value 0.516 


ICC (SE) 0.197 
(0.061) 
Variance school level (SE) 9.34 
(3.54) 
Variance pupil level (SE) 38.05 
(2.35) 
Total sample size (schools) 545 (19) 
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Table F4a: Analysis of primary outcome, literacy—intervention (singing) versus control (drama), 
FSM pupils only 
Unadjusted analysis Adjusted analysis 


-0.03 
Regression -1.41 -0.31 
coefficients (95% Cl) (-2.67 to 2.06) 


Effect Size (Cl) 


P-value 0.431 0.799 


ICC (SE) 0.027 0.000 
Variance school 2.15 0.00 


Variance pupil level 77.90 31.31 
(SE) 


(11.48) (4.45) 


Total sample size 99 (15) 99 (15) 
(schools) 


Table F4b: Analysis of primary outcome, literacy—intervention (singing) versus control (drama), 
FSM interaction model 


Adjusted analysis 


Regression coefficient of the interaction between FSM 1.73 
i 0, 
status and experimental cell (95% Cl) (-0.68 to 4.13) 


P-value 0.160 


ICC (SE) 0.006 
(0.014) 
Variance school level (SE) 0.18 
(0.41) 
Variance pupil level (SE) 29.52 
(1.82) 
Total sample size (schools) 545 (19) 
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Table F5a: Analysis of primary outcome, maths—intervention (music) versus control (drama), FSM 
pupils only 


Unadjusted analysis Adjusted analysis 


0.02 
Regression 0.32 0.22 
coefficients (95% Cl) (-1.91 to 2.35) 


P-value 0.853 0.836 


Effect Size (Cl) 


ICC (SE) 0.118 0.194 
Variance school 12.92 8.34 


Variance pupil level 96.28 34.61 
(SE) 


(11.66) (4.62) 


Total sample size 151 (15) 151 (15) 
(schools) 


Table F5b: Analysis of primary outcome, maths—intervention (music) versus control (drama), FSM 
interaction model 


Adjusted analysis 
Regression coefficient of the interaction between FSM 0.16 
i 0, 
status and experimental cell (95% Cl) (-2.32 to 2.56) 


P-value 0.894 


ICC (SE) 0.208 
(0.060) 
Variance school level (SE) 10.20 
(3.66) 
Variance pupil level (SE) 38.95 
(1.95) 
Total sample size (schools) 814 (19) 
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Table F6éa: Analysis of primary outcome, literacy—intervention (music) versus control (drama), 
FSM pupils only 


Unadjusted analysis Adjusted analysis 


0.24 
(-2.73 to 3.20) 


0.875 


0.007 
(0.034) 


Effect Size (Cl) Cannot estimate. Model does not 


converge 


Regression 
coefficients (95% Cl) 


ICC (SE) 


Variance school 


0.54 
level (SE) (2.53) 
Variance pupil level 74,51 
(SE) (8.87) 


Total sample size 151 (15) 
(schools) 


Table F6b: Analysis of primary outcome, literacy—intervention (music) versus control (drama), 
FSM interaction model 


Adjusted analysis 


Regression coefficient of the interaction between FSM -1.10 
i 0, 
status and experimental cell (95% Cl) (-3.12 to 0.93) 


P-value 0.288 

ICC (SE) 0.027 
(0.016) 

Variance school level (SE) 0.78 


(0.48) 
Variance pupil level (SE) 28.00 
(1.40) 

Total sample size (schools) 814 (19) 
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Appendix 7: Memorandum of understanding with schools 


Gucaltior 


| nadowment ative 
Foundation tures 


J Leading education 
Nat Cen ‘e7 and social research 
Social Research that works for society institute of Education 


University of London 


Act Sing Play: Investigating the impact of arts provision on achievement 


INFORMATION PACK 

Thank you for your interest in taking part in our study to see if arts provision has an effect on children’s 
achievement. This letter explains more about the project, including what you can expect to happen once 
you have confirmed your school’s participation. 


Creative Futures, with funding from the Education Endowment Foundation (EEF), is conducting a year- 
long study to assess the impact of a structured music and drtama programme on the achievements of 
Year 2 pupils. We will be working with research teams from NatCen Social Research (NatCen) and the 
Institute of Education (loE) to evaluate the programme. 


The study is a randomised control trial so we will put Year 2 pupils into either musical instrument, singing 
or drama classes by a random method and once they are in a group, they cannot change. Children will 
take part in the assigned workshops for one academic year starting in September 2013, and finishing in 
June 2014. To evaluate the programme, it is important that we test the educational attainment of all 
participating pupils before and afterwards. This will allow researchers to assess the impact of the various 
activities on children’s achievements and is a vital part of the programme. Members of the research 
team will also visit a small number of schools to observe the workshops and talk to staff about how the 
programme is being delivered in their school. 


If you agree for your school to take part in the programme, we will provide letters for you to send to 
parents explaining what the study involves. This will give the parents an opportunity to opt their child 
out of the programme — although we hope that with your help no parents will decide to do this. 


There is more detailed information about what to expect from taking part in this programme in the 
following pages of this letter. We very much hope that your school is able to take part in this important 
study which will help provide a valuable insight into how high quality music tuition may help children do 
better at school. 


Please sign and return the attached form to me as soon as you can and no later than by Friday 10 May 
giving permission for your school to take part. You need only send p.6 of this form to me — the page with 
your signature — and this can be sent to me as a scanned file by email, or by post. We really hope you will 
confirm your involvement in this exciting project. 


Meanwhile, if you have any questions about any aspect of the project please contact me. 
Yours sincerely, 

Julian Knight 

Creative Director, Creative Futures (UK) Limited 


Office 9a, the Beethoven Centre, Third Avenue, London W10 4JL 
T: 020 8964 2700 M: 07980 820323 E: julian@creativefuturesuk.com 
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@How will pupils be allocated to the three different activities? 


Each Year 2 class involved will be randomly split into three equal groups of up to ten pupils. One group 
will learn a string instrument (predominantly violin, with some violas and cellos offered too); the second 
group will take part in a singing and vocal programme; and the third group will have drama lessons. Each 
group will do their activities weekly during the school day and will receive around 32 workshops over the 
academic year. The workshops will last for 45-50 minutes. 


@ Is it possible for pupils to choose which activity they are allocated to? 


No, because pupils will be randomly allocated to the different activities. This study is a randomised 
control trial which means that children can't choose what they want to study. We will put children into 
either musical instrument, singing or drama classes by a random method and once they are in the group, 
they can't change - this is critical to the design of the study to make sure that the approach is scientific 
and not biased by the choices of children or their parents. 


@ Why is it important for pupils to be randomly allocated to workshops? 


Random allocation is essential to the evaluation as it is the only way that we can say for sure what the 
effect of the intervention is on children’s attainment. We hope that schools involved will support this 
and urge all parents of pupils in Year 2 to agree to the terms of this project. 


© When will the project start and end? 
Workshops will take place during the school day, starting in September 2013 and finishing in June 2014. 
@ Who will be delivering the workshops? 


Creative Futures will be working closely with the Schools Music Service in each area, and many of the 
vocal and strings tutors for this project will be drawn from the pool of tutors already engaged by your 
local Music Service. Drama tutors are likely to be engaged directly by Creative Futures. All tutors will hold 
a valid CRB/DBS disclosure. 


@ How will the workshops be organised? 


Julian Knight from Creative Futures will liaise with your school to organise the workshops. All three 
activities will probably need to occur simultaneously in different rooms, so this may have implications on 
space and timetabling in your school. Tutor availability will be a factor too, and we will be grateful for 
any flexibility schools can bring to the timetabling. Once the timetable is set, we would ask that it be 
maintained for the whole year, and that any known events, trips or exams (etc.) that may cause 
disruption to planned workshops are identified as early as possible, so that alternative workshop times 
can be arranged if necessary. 


@ What will children learn in the workshops? 


We have devised a programme of study for each of the three groups (string instrument, vocal, drama), in 
order to ensure consistency across all 19 schools taking part in this study. The workshops will be 
instructive as well as fun, and are tailored to the needs and requirements of Year 2 pupils. The string 
instrument lessons will assume that all the pupils are beginners and will take them through the first 
stages of learning the violin, viola or cello; the vocal curriculum is based on the Kodaly Method and will 
give a thorough musical grounding through rhythm and singing; and the drama curriculum will develop 
the imagination through story-telling. 
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@ Will musical instruments be provided? 


We will provide string instruments for all pupils allocated to the string lesson group. It may be possible to 
choose between violin, viola and cello, although the majority of pupils will be learning to play the violin. 


© Can pupils take the instruments home? 


Yes, we encourage this so that they can practice if they want to — as long as they bring them back to 
school each week for the workshop. 


© Will pupils be able to keep the musical instruments after the project has ended? 


The string instruments will be provided on loan, so we will want them back eventually. As the 
instruments are likely to be less than full-size, they will not last the pupils for ever. If pupils want to hold 
on to their instrument for a period of up to one year after the project finishes in order to continue 
learning (eg through the Music Service), we will extend the instrument loan accordingly. Most of the 
instruments will be transferred to your local Music Service after the project, so that other pupils in your 
area can use them in future. 


@ Will all children be able to take part in the programme? 


This is an inclusive programme, and we would like all Year 2 pupils, including those with Special 
Educational Needs (SEN), to take part. In the case of some statemented pupils only, where their specific 
situation means that they are unable to participate in particular activities, we will consult with you and 
allocate them to the activity that is most suitable for them. This is the only situation in which a pupil 
would be allocated to an activity outside the random allocation. 


@ What sort of research activities will children and school staff be taking part in, in addition to the 
workshops? 


NatCen and the loE will carry out two rounds of pupil testing - one round before the music programme 
starts and one round after it ends. Tests will be administered to all pupils taking part in the programme. 


Researchers will also monitor how the project is working in schools, including through: profiling and 
monitoring the attendance of participating pupils; a short online school survey; and carrying out case 
studies of some of the participating schools which will consist of short interviews with key members of 
staff and observations of some workshops. 


@ Why do you need to carry out pupil testing? 


As this project is primarily a research project, we need to test children before they start the programme 
of workshops and at the end of the programme. We will combine results from the 19 schools taking part 
in this study in the analysis and will compare how well children in the three groups (string instrument, 
singing and drama) did at the end of the programme in terms of their numerical and literacy skills. 


@ When will testing be done? 


Tests of numerical and literacy skills will be carried out by NatCen in June-July 2013, when pupils are in 
Year 1. Any new pupils who join the school in September 2013 will take the tests at the beginning of that 
term. All participating pupils will then be tested again at the end of the programme in June-July 2014. 


Researchers from the Institute of Education will carry out tests around musical skills and self-efficacy. 


These will take place once the pupils have been allocated to their respective activities in September 
2013, and in May/June 2014. 
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©@ How will the tests be administered? 


NatCen interviewers will administer the paper and pencil tests in small groups of 5 or 6 children. The 
interviewers will be fully trained in how to administer the tests to children. They will liaise with your 
school to arrange a convenient time to visit and carry out the tests. NatCen interviewers working on this 
project are all CRB checked. 


Researchers from the loE will work alongside class teachers to complete a simple ‘smiley face’ 
questionnaire. All pupils will complete this element of the research, working individually (with support 
where needed) within the classroom setting. In addition, researchers from the IoE will assess the singing 
competency of those pupils allocated to the singing programme. Working in groups of 4 or 5, the pupils 
will work with the researcher to complete a series of simple vocal tasks such as singing a well-known 
melody and copying short extracts sung by the researcher. The researchers are specialists in music 
education research who will work with schools to create an enjoyable experience for pupils. All 
researchers are CRB/DBS cleared. Finally, the tutors for those pupils allocated to the string programme 
will complete simple end of term records of achievement. 


© What tests will you be using? 


NatCen will be using the reading and maths modules of the Performance Indicators in Primary Schools 
(PIPS) which are produced by Durham University’s Centre for Evaluation and Monitoring (CEM). Each of 
these two modules will be administered on different days and take up to 30 minutes to complete with 


pupils. 


The loE will use a pupil questionnaire and singing assessment based on research tools established in the 
research evaluation of the National Singing Programme (see imerc.org for further details). The pupil 
questionnaire will take up to 30 minutes to complete. The singing assessment will take up to 12 minutes 
to complete (as part of a group of 4-5 pupils). 


@ What is involved in the observations of workshops and case studies: 


NatCen and the Institute of Education researchers will be observing some workshops over the course of 
the year and developing qualitative case studies based on some of the schools. 5 schools will be selected 
by NatCen and 5 schools by the loE. 


NatCen’s researchers will observe two sessions (one of a stringed instrument and one of a singing lesson) 
and carry out four 30-45 minute interviews in each of the case study schools with key members of staff 
involved in the delivery of the music workshops as well as a member of staff who has oversight of music 
tuition in the school. loE researchers will also attend two sessions (one stringed instrument and one of a 
singing lesson) in order to observe best practice in the teaching of these subjects. 


If your school has been chosen for these observations or case studies, we will liaise with you about this 
separately to arrange a convenient time to visit. 


© What will parents be told about the music programme? 


Parental permission will be sought for all pupils involved in the project, and we very much hope that, 
with the school’s help, all parents will agree to their children taking part in this programme. 


We will give you letters to send to parents in participating classes explaining what the programme 
involves. Parents will be given an opportunity to ‘opt out’ of the music programme by letting your school 
know if they do not wish their child to take part. However, it will not be possible for parents to opt out of 
the testing and still receive the workshops. In other words, if children want to take part in the project, 
they will also need to take part in the testing phases of the programme too. 
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© How will teachers be involved in the programme? 


NatCen interviewers will ask class teachers to provide some background information about the 
participating pupils to help provide some context to the analysis we do. This information will be the 
child’s Unique Pupil Reference Number, gender, date of birth, eligibility for Free School Meals, ethnicity 
and SEN status for each pupil involved in the music programme. This will be explained in the letter to 
parents. 


@ What about data and confidentiality? 


Pupil’s test responses, and other data requested from the school, will be collected and accessed by the 
research teams at NatCen Social Research and the loE, and will be treated in the strictest confidence and 
in line with the Data Protection Act. With the permission of the Department for Education, named data 
will be matched with the National Pupil Database and shared with NatCen Social Research and the EEF 
for research purposes only. 


© How will the data be used? 


The data from this study will be used to prepare research reports for publication. No individual schools 
or pupils will be identified in any reports arising from this research. Data from each of the 19 schools 
involved in the project will be analysed anonymously, and findings will be reported anonymously, for 
example “x % of children had numerical skills at x level or above”, so it will not be possible to identify 
individual pupils or schools. 


@ Will there be a report on the project’s findings? 


Published reports with research findings are due to be available at the end of 2014, and we will be very 
happy to share our findings with the schools involved. 
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Appendix 8: Security classification of trial findings 


2. Power: 5. Threats to 
1.Design: What is 3. Attrition: 4. Balance: D validaee 0 
What is the ee What is the Adjustment vaticlty: 
d minimum s Adjustment to 
quality of the AV-) Me} lel ge) 8) to rating to ; 
(ol-3a-el ¢-] 9) (= account for issues 
design of the out from the account for 
aluation? vil ety evaluation? balance? tik 
I t Vv f f 
hii start? nh interpretation? 
Evaluation design Implementation Analysis and interpretation 


1. Design 2. Power 3. Attrition 4. Balance 5. Threats to 
{L259} validity 
5a Well-balanced on as 
abseivables No threats to validity 


Fair and clear experimental 
design (RCT, RDD) 


Well-matched comparison 
(quasi-experiment) 


Matched comparison 
(quasi-experiment) 


Comparison group with 
poor or no matching 


No comparator Imbalanced on 
, observables 


Significant threats 


The final security rating for this trial is 4 @. This means that the conclusions have moderate to high 
security. 


This evaluation was designed as a randomised controlled trial, and the sample size was designed to 
detect a minimum effect size of less than 0.2. There was low to moderate attrition (11%), reducing the 
number of padlocks to 4 @. There was no indication of imbalance at baseline. The post-tests were 
administered by the evaluators and marked by an independent testing company. Therefore, there 
were no substantial threats to validity and the final security rating is 4 @. 
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Appendix 9: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found on the 
EEF website. Cost ratings are awarded as follows: 


Cost rating Description 

£ Very low: less than £80 per pupil per year. 

EE Low: up to about £200 per pupil per year. 

ELE Moderate: up to about £700 per pupil per 
year. 

EEEL High: up to £1,200 per pupil per year. 

ELEEELE Very high: over £1,200 per pupil per year. 
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